Why most product planning is bad and what to do about it
TL;DR: We tried OKRs, they created more ceremony than clarity. Our solution: Problem Driven Development, a 4-day quarterly process focused on identifying problems (not solutions), prioritizing as a team, and committing publicly. It's kept us shipping at velocity even as we've scaled to 1.7M+ users.
For most of my friends and colleagues at mature software companies, there are usually three ways for an item of work to get put on the board to eventually be done.
There is a giant ceremony that determines what gets done by a grab bag of metrics. A deal gets blocked by a missing feature, and the engineering team scrambles jets to eliminate the blocker. Founder feels like we have to build something.
Thats not to say that every company is a disorganized mess or a bureaucratic hell scape but, I have never met any engineer who said: “Wow, I just love my company planning process.”
These words are seldom spoken in the english language.
Railway, was fast approaching 1. and 2. at the same time.
Despite us using excellent tools (shoutout Linear) planning is much as a cultural phenomena as well as an interesting engineering problem. From our perspective, we would finalize the features and the requirements what we would want to build once every 3 months and then at times get blindsided every now and then from a new business priority or an incident. Especially now serving 1.7M+ customers, we have to aggregate our taste, feedback, and opportunities and make a roadmap that will get us everything we ever wanted.
This was an issue.
Why write about something that you would read in Rand’s? It’s Q4 for those who celebrate, and we felt that reigniting the agile vs. waterfall armistice needed to be torn up. (Besides, we’re blogging like it’s 2005.)
…and we have spent the better part of 18 months to improve our planning so that we finally get to a process that is not bad, and if you’d like, you can steal it, so that you can deliver excellent products.
Mike Tyson asked about his fight plan against Evander Holyfield responded with; “Everyone has a plan until they get punched in the mouth.”
And that quote ever since would be used out of context to fight the implementation of Objectives, Key Results across the world. At Railway, we aren’t Anti-Planning, we are Anti-Bad-Planning and as such, we used to avoid a lot of it because we always felt that a bad process is massively negative to the status quo. However, we did get to the point where we needed to finally harness the collective attention spans of the company and work towards a goal.
Railway at the time was (and is) a vision led company. We fundamentally believe that most infrastructure boilerplate shouldn’t exist so that you can work on what matters. Back in ‘21, we went from a world where we could do our work in weeks, to work that required us to think in months. I implemented a somewhat lighter form of SAFE which splits work between “sprints”, which were short pieces of work, then initiatives which helped grouped “epics” which are long running work-streams to deliver a feature.
Despite us having a strong founder, we still wanted the ability to have employees bring projects that they would be excited to work on.
As with most young companies, we found quickly that new work would come in and unseat the old work and we would begin anew.
So then we turned to OKRs.
We thought that having some long term objectives will help us focus the company.
We implemented them faithfully, we all read the John Doer book. …and it worked… somewhat.
We can make this a whole blog post on OKRs. But, trying to hold back here from just ranting and to give you the relevant information.
OKRs work REALLY well when you have concrete goals and really simple ways to measure them. Its why they belong at their primary birthplace, the factory. Objectives work when you have something binary or “limited”, like a product existing… or not. Or a product meeting a benchmark… or not. It’s easy to rally a team around them and conquer the world.
Where OKRs start to falter, are when you need to use them to prioritize work to meet a “unlimited” objective. Like: “Increase conversion rate of landing page”
OKRheads will notice that’s a particularly weak objective, however, teaching an entire organization to all of a sudden be great goal setters, which is what OKRs require you to do made it difficult for engineers to 1) bring them and then 2) plan work around them. Which leads to the first big issue of OKRs- which is that it really depends on the human psychology of the team. Engineers straddle the line between concrete numbers such as uptime, and more creative endeavors such as figuring out how to get a feature implemented right. Whenever the work enters the creative realm, the wheels come off.
Which is why outside of the factory, OKRs are great for Sales. You set a number, you get alignment on hitting that number, and then you pick up the phone, email, or LinkedIn until you hit it. The psychology of an account executive matches the planning process.
Where OKRs work great, is for alignment. If you are able to set great goals and if you wanted to prescribe a bunch of work toward said goal. …and most startups reach for the High Output Management book (which I love) because there is the lack of alignment between different orgs at a startup. Which is to say, I don’t shame any company reaching for them, the same way newborns have the palmar grasp reflex. We yearn for the alignment.
The second big issue for OKRs is that they are, by design, inflexible, once you commit to them. If you spend a week or two planning for them for the year or quarter, and you are midway through realizing that the “KR” part is incorrect, thats a one way door.
So we felt pressure into making sure that the OKRs were indeed correct. Which is where you enter the issues of the performative aspects of planning. It felt “mature” for Railway to be having these discussions, even though… it wasn’t productive.
At Railway, we would spend a significant amount of time discussing what is a valid OKR, only for us to realize we were two days into the planning process and we’ve yet to decide what we should build for the OKR.
This is when Christian, our Head of Operations had an answer for us.
Instead of crowd sourcing the OKRs from the company and bubbling them up per function.
Jake and Christian would work on some top level guidance that would provide a macro view of our finances, priorities, and strategy that we have to keep in mind. We then have a Hex dashboard with topline metrics from different sides of the product and business. From a GTM perspective, it’s engagement, signups, and revenue. From an Engineering perspective, it’s support tickets, uptime, and feature performance.
This data would give us a “theme”. (You may have seen them in our launch weeks.)
Whenever someone goes on Central Station and requests a feature, depending on how large the work is, that usually determines if that request becomes a “Project”.
Then around the start of 2024, Christian spun up a Notion DB, and punched in a simple template for each new entry.
Railway at this point and time was organized into three sections of the business. Product Engineering which delivers features and value to our customers via the UI and terminal. Platform which supports the product with the Infrastructure and the APIs to control that Infrastructure. Then Logistics, which is a synthetic team which includes the Support, Marketing, and Sales function working to be the voice of the customer.
We would then fill out a “Project Candidate” and then go to bat on trying to figure out if was a priority that tied to the “theme” that was presented.
If something is a P0: it’s an existential company risk, we MUST deliver it.
Then, a P1: something we need to deliver for this quarter
Lastly, P2: a nice to have.
This worked well until some cracks formed.
At the start of this new process, our team was small enough to list maybe 50 project candidates, discuss as a group and then be on with it but with 200. Spending all day on a call with your co-workers to discuss if something was worth it isn’t the best use of time.
Second, Project Candidates would sometimes have full on fleshed out RFC style sections where the author would have a solution ready to go. When a candidate got deprioritized, it’s understandable why someone would feel not great about their hard effort not being reciprocated.
Third, to deal with the larger amount of project candidates and additional sources of project candidates, we would set up additional meetings before the planning week that would eat into our engineering cycles. Not that we want to squeeze every engineer for what they are worth, but these negotiation calls would take longer than the main call sometimes. Worse even, was that we would completely reset the board quarter after quarter.
Lastly, which was starting to be a bigger issue, was that having Project Candidates being a bottom up process was great when the people who knew that work they needed to do can fit the whole work-stream in their head. But… we were about to embark on a multi-quarter effort that would span all parts of the business to deliver Railway Metal. This system was not great for uncovering planning gaps that required attention from a different team.
(With that said, we did ship some really good software thanks to this planning process.)
However, Jake, the team, aren’t ones to sit on their laurels so we refined it until we have our current process.
So we flipped the project system on it’s head.
Instead, we practice what I call: “Problem Driven Development”
Rather than spending a loaded amount of time picking feedback items from our feedback systems, and spending a loaded amount of time on trying to flesh out requirements. We collect problems on a continuous process that begins in earnest on the Friday before our planning week.
We stopped asking people to propose solutions and started asking them to articulate problems. No more half-baked RFCs that people felt attached to. No more solution-first thinking that locked us into approaches before we understood what we were solving. Just clear problem statements.
Here's what a problem looks like in our system:
Problem Title: "Users can't debug failed deployments without SSHing into containers"
Notice what's missing? Any mention of how we'd solve it. That comes later, after we've committed to solving the problem. Then we turned the two week ceremony into a lightning 4 day sprint.
After the problems are listed and filled out fully, each team on Day 1 enters problems on their own. If a problem isn't fleshed out, it's put in the Parking Lot and we kindly nudge the idea person to fill the template.
By the time we get into a room together, everyone's had time to process what's on the board.
Then on Day 2, the planning captain will have a closed session with the team (with spectators from other teams) to put a best guess priority from P0, P1, and P2. If we find that a problem is contentious or requires an external dependency, we mark it so the day after, we can discuss it.
By having each team prioritize independently first, we avoid the tragedy of the commons where everything becomes P0 because someone shouted loudest on the call. Platform can look at their problems and say "yes, API reliability is more important than API versioning right now" without having to negotiate with Product in real-time.
(We have spent countless hours talking about difficult questions on a call when we have been able to diffuse rough conversations ahead of time.)
On Day 3, we get to a 95 percent certainty on the priorities listed and tie break the dependencies. We also confirm that we have the capacity and staffing to deal with the problems listed on the board. If not, we add a problem for hiring for a specific role.
This is where cross-team dependencies surface naturally. When Platform marks "Multi-mount volumes" as P1 and Product marks "HA DBs" as P1, we can see that one blocks the other. We're not discovering this mid-quarter when someone's already three weeks into building the wrong thing.
The capacity check is crucial too. We look at people's current commitments, oncall rotations, and whether anyone's about to take parental leave. If we have eight P1 problems and capacity for five, we have an honest conversation about what drops to P2 or what we need to hire for.
Then lastly, Day 4, commitment. Everyone looks at the whole list, mention final objections if we should be working on something that we aren't— or vice versa. After we confirm priorities, we assign problems to people by giving them a DRI (Directly Responsible Individual).
Then, we look into each other's eyes, as much as anyone can on a video call, and commit.
After the starting gun is off, we then write RFCs on problems to see how we can best solve them which eventually become Linear tickets.
We're working on open-sourcing our Notion templates because honestly, this isn't rocket science. It's just:
Cataloguing your problems, not solutions Let teams prioritize independently before negotiating Front-load the hard conversations Commit publicly Then, and only then , figure out how to solve it
But the real hard part was building a culture where people feel safe bringing up problems without having to defend their pet solutions.
Keep process to a minimum, focus on performance of shipping, not performative work.
And like any good product, our planning process will keep evolving. Maybe in six months we'll realize Problem Driven Development has its own cracks. Theres always plenty to do on planning and I think we generally know how it’ll need to evolve, but we try to take small steps from cycle to cycle. If that happens, we'll write another blog post about what we learned and what we changed.
Until then, we're shipping.