Stupid Baboons, Stubborn Elephants: A Product Engineer’s Guide to Working with Platform Engineers
A while ago I was telling a friend how I was frustrated at work. I had been adding support for a really important product feature, and our platform teams were making it impossibly difficult. It felt like every line of code required a month of negotiations to get approved. Let’s be honest here — I wasn’t telling, I was complaining. “Do they think we’re idiots?!”, I asked.
She was smiling with what I thought was empathy, until I was done and she said: “You are idiots”. Then she proceeded to explain that from her experience as a platform engineering manager, product engineers and platform teams live in a parallel universe. Finding common ground is quite a challenge. I believe at one point she used the words “stupid baboons” (which gave this post its title).
My jaw dropped. I don’t consider myself to be inherently stupid (please avoid sharing evidence to the contrary in the comments section), but was I acting stupid? Things certainly weren’t going smoothly. What was I missing here?
I spent some time listening to her, then asked for some guidance from more senior members of our group, and did some thinking about where things did go well as opposed to where there was most friction, and that’s where this post came from.
But before I dive into the perspective I’ve gained about working with stubborn elephants, AKA platform engineers, let’s clarify what we’re discussing.
Product vs Platform vs Fullstack
My definition of “platform engineer” is an engineer whose role is to provide a framework for other developers to use , usually internally, whereas a “product engineer” is an engineer whose role is to develop user facing features.
As a rule of thumb, if you’re doing the agile thing where you word tasks as user stories, and the user in the story is, well, an actual user — you’re probably a product engineer. If the story is about a developer in your company, that’s a good sign you’re a platform engineer.
I’ve also seen some misconceptions about what a fullstack engineer is, some confusing them with product engineers. The definitions I’ve seen seem to span a wide range: from “a genius who can do anything” to “a garbage can with access to a keyboard”.
My definition of fullstack is simple: a developer who can complete a programming task which includes backend and frontend work. Seniority or expertise is irrelevant. Fullstack can be a tic-tac-toe react app with a firebase backend for keeping score, or an app tracking health stats storing your data on a serverless blockchain (or something else with extra buzzwords to make it sound complicated to implement).
You’re doing frontend + backend? Congratulations! You’re a fullstack developer!
I concede that most fullstack engineers are product engineers, in that they’re developing features for users, but they’re not synonymous. Depending on the organizational structure:
- Product engineering work may be split between backend and frontend engineers.
- Fullstack engineers develop internal tools like insights, analytics, and quality control.
- And finally, even though they’re frontend engineers I don’t think you can seriously suggest that the React core team is doing product work.
So, now that we’re clear on the terms — let’s talk about where the difference between product engineers and platform engineers comes into full play.
Perspective
Product engineers are focused on user facing features and customer needs, viewing platform code as a set of tools and services which should allow product features to be written quickly and efficiently.
Platform engineers are focused on creating an infrastructure which can be used by many different consumers for a long time, product code’s little quirks and special cases interfere with the ability of platform to provide flexible and robust services.
In addition, product engineers usually work on a set of features in the same area, perhaps switching teams every so often. Therefore, their perspective is mostly limited to what they’re working on right now (or a few things they’ve worked on in the past) and the platform services they use. They don’t have much knowledge and awareness of what other product teams are doing or how their work might affect them, but are deeply aware of how different platforms behave and how their different paradigms and conventions affect product code.
Platform engineers may have the same blind spots as product engineers regarding what other platform teams are doing, but just by their role in the stack they can be aware of how certain changes might affect other teams doing platform or product work. As my friend put it: “You may need a 2 in this parameter, but this other team — they need it to be a 3. We can’t support both, you’ll have to figure it out yourselves”.
Speed vs. Quality
If you’re an engineer of any type in 2019, you’re probably working with agile. If you’re doing it “by the book”, you should be working in short sprints. If your task doesn’t fit in the sprint — you break it down into smaller pieces to make sure it gets done within that sprint.
Only, this is a bit of a lie: Tasks are not always entire features. The fact that we got something into production doesn’t mean the feature is done. It’s common to release incomplete features which won’t be ready for months, so we gate behind feature flags until we’re actually ready.
Still, product projects are usually short — planned for release within a few months at most. Plan, code, test, release — or you’ll miss the market! Platform projects are long, their planning is thorough and slow, and some projects take years to complete.
Now, I like clean code as much as the next person (though I believe design pattern overuse should be considered one of the 7 deadly sins), but as a product engineer — if I’m stuck with a tangled mess, I have to plow through it and get stuff done, fast. I can’t stop everything and clean up. As much as I would like to, I have a deadline.
In addition, product requirements change constantly, some features being killed off or modified extensively, meaning that planning too far ahead and over-engineering code is an absolute waste of time, a classic case of YAGNI.
This makes product engineers look bad, as if we’re hacking stuff together with chewing gum and a thumbtack, whatever — as long as it works.
Undead Code (AKA Legacy/Deprecated)
A side effect of platform projects taking so long is having to deal with legacy code. When a product feature dies, most of its code can be safely deleted. No one does this because it slips between the cracks or because they’re too scared (for some understandable reasons), but in general — most code related to a dead feature is just sitting somewhere being dead, not bothering anyone who needs to write new features.
Platform code doesn’t behave that way. It hangs around in zombie state making all kinds of trouble for product engineers. As far as platform is concerned, it’s deprecated and shouldn’t be supported. But if there are still features still under development which are using legacy code — they need support and sometimes modification. Every once in a while there’s a feature which requires a big foundational change, and then it’s Code World War Z all over the place.
No one likes messing with legacy code, but what choice do product engineers have? Remember, we have a deadline, soon. We can’t wait for two-three years until that deprecated zombie code is dead and buried. Often the effort of switching feature code to use whatever platform is supporting is far too great and doesn’t fit in the product timelines.
Distance
If you work in a tiny startup where everyone sits in the same room and there is no real separation between engineering functions — you can skip to the next section. However, if you’re working in a more mature company there are usually different teams and even entire engineering orgs for product vs platform work. This causes a mental/emotional distance, as well as physical distance between the various functions.
Hopefully, the company you work for doesn’t encourage an “us vs. them” attitude, but that type of thing can develop organically if it’s not actively acknowledged and addressed.
If you’re located in different buildings or different cities across the globe, it makes all of this much worse: When you don’t have frequent professional or casual contact, it’s much easier for to view each other as “stupid baboons” or “stubborn elephants”.
Why can’t we all just get along?
So far we’ve seen 4 major friction areas:
- Perspective: Product and platform engineers have different ideas on what’s important, causing them to view effort in certain areas as a nuisance or as the highest priority, depending on their point of view. They also have different perspectives on the system as a whole — cross team concerns, APIs etc. which cause friction and complications in unexpected places.
- Speed vs. quality: Product engineers are working on shorter timelines and with constantly changing requirements. Putting an emphasis on speed and less on long term engineering makes them appear not to care about code quality.
- Zombie code: Product engineers sometimes need to modify legacy platform code, waking undead code from its resting place and making trouble and extra work for platform engineers.
- Distance: “Us vs. them” attitudes and lack of a solid working relationship makes it harder to see past the “stupid baboons” and “stubborn elephants” labels and respect each other.
So, what can we do to work together better?
Communicate! Sorry for the cliché but really, that’s pretty much the only solution.
Clarity
While planning, be clear about product timelines and constraints. Do not be implicit, explicitly state that you are consciously making trade-offs and what they are.
Plan ahead
Have a long term plan, share it with other teams, and use it to make short term decisions with that long term vision in mind. This is more efficient from a collaboration point of view, enabling other teams to understand the big picture and come to agreements once, instead of having to context-switch every time an issue comes up. It’s also more efficient from a development point of view, as small decisions are easier to make when the path ahead is clear.
Respect their time
Platform is not waiting around for you to give them more things to do, they’re quite busy on their own. If you need their help, give them them courtesy of a heads up letting them know what your plans are and what kind of support you’ll need.
Respect them personally
Be respectful, assume not only good intentions, but also competence. After all, we’re all working for the same company, hopefully towards the same goal. Even if our immediate interests may clash, we should make an effort to look past them.
Ask for help
Remember, our perspective is limited, we certainly don’t know everything. Asking for advice from other teams is nothing to be ashamed of, in fact — it’s the exact opposite, it’s an essential part of getting our work done at the highest quality possible under those pesky time constraints.
Build a relationship
If product and platform engineers talked a bit during happy hour or stood around meeting rooms waiting passive-aggressively outside a conference room for the previous meeting to end every once in a while, or in general — had a solid relationship and a wider personal perspective of the people you’re working with, you might think they’re not so bad after all. I don’t believe in forced offsite activities for building team work, it has to be done on a daily basis. Find the right opportunities and make it happen.
Communicate directly
Don’t be too afraid (or lazy…) to communicate directly. Get out from behind your computer and into a room. Talk. Meetings can be annoying and distract you from “real work”, but sometimes they are the most effective way of communicating and absolutely necessary in order to get things done smoothly.
- If you don’t work in the same site and don’t have the budget for travel for every little feature (which makes sense, after all), pick up the video conference app of your choice and talk to each other anyway.
- Slack is also not bad for building relationships, as it feels more personal and casual then emails.
- Use lots of emoji. I have no data to back this up but I’m sure it helps.
- Stay away from JIRA or other task management systems. Those are important for keeping track of project status, but terrible for building trust and communicating. Update JIRA after you’ve communicated directly and verified you’re all on the same page.
These guiding principals will (1) make product engineers look less like “stupid baboons” and more like the highly skilled professionals that we are, and (2) help us write better, cleaner, more maintainable code. Everybody wins!
Thanks for reading this far, I hope you find my insights useful. Remember, this was written from a product engineer’s perspective — I’m sure there are many useful insights from platform engineers which could have been added here. I’d be more than happy for any of you to write a similar post from a platform engineer’s point of view, or just leave a comment below.
Special thanks to:
- Karen Cohen for the inspiration, awesome conversations, and feedback on this post.
- Alon Kiriati and David Goldstein for their excellent comments and insights.