This week we sit down with Teresa Dietrich, Stack Overflow's new Chief Product Officer. Teresa, a CMU grad and tech industry veteran, shares a story from her time at AOL: the day three little dots emerged without warning and turned their network upside down.
At the time, AOL was one of the largest providers of dial-up internet access and its messaging platform was amongst the biggest in the world. As Teresa tells it:
My first job out of college was at AOL during the height of the first Internet rush. I was in technology operations, first as a network engineer, then eventually reaching senior network architect. That job helped shape my perspective and the way I think about problems, both in how I architect, but also troubleshoot.
At the time, I had no idea what I was a part of. The scale of problems that we saw was just amazing to me. Just two years out of college, and I was actually helping to keep CNN.COM up on September 11th, 12th, and 13th.
I was also there when AIM launched the three dots. When I’m on my iPhone or Slack and I see dot, dot, dot when someone is typing, sometimes I have a little PTSD thinking back to the launch. Because we had no idea how much chaos that would cause.
This was a random day in the middle of the week. All of a sudden, we started getting these alerts of router crashes throughout our data center, throughout multiple data centers. We all jump up and look at the logs to see what the router was doing and looking at bandwidth graphs. The bandwidth hadn't gone up at all. There was no discernible change from before they started crashing to when they started crashing.
So we're trying to look at the data we don’t look at as often. We're calling into our data center techs, getting sniffers plugged into some of the routers where there's crashes going on and trying to figure out what is going on. I mean, it was literally like every 30 seconds, another router was falling over. So we'd get it back up. And it worked for awhile. We were like, are there gremlins in the data center? What's going on here? We finally got the sniffers plugged in and we're starting to sniff traffic. We started seeing these 60 byte packets being sent from the AIM servers.
Those of us who are also AIM users, some of us said, you know, it's funny they launched that “Your buddy is typing” indicator today. We noticed that happening for those of us who updated our clients. What we realized is they were sending an awful lot of packets. “Your buddy is typing...” “Your buddy has stopped typing.” “Your buddy is typing..."
At some level, we knew this was a limitation of the hardware. In talking to Cisco or Juniper we knew the packet per second limitation of our routers, but we never worried about it. We weren't tracking that.
You never track and collect and graph a metric until you realize you need it. So we literally had to call over to the AIM development team and say, “Shut it down and let's work together and figure out how we can implement this feature without sending an insane number of 60 byte packets and taking down large swaths of the network." It ended up shifting to a state change check and eventually we had millions and millions of concurrent users enjoying the feature on a daily basis.
Along with a great story of her time at AOL, we also chat about what the job of a chief product officer is today. At a place like Stack Overflow, how do you unite functional departments across the company - from marketing to sales to engineering. How do you figure out the right incentives, so that the data you're measuring against is aligned with the long term health of the company and the community?
"I don't focus on shipping, I focus on impact," Teresa told us. "That's where product management, engineering, and design come together. Product management is focused on value. Engineering is focused on quality, and design is focused on usability. If you think of that as Venn diagram, impact is where those three things overlap and happen."
Lastly, we chat about the incredible velocity with which new coding languages and development frameworks emerge in the tech industry. Teresa shares her philosophy for encouraging an engineering team to level up and learn new skills while ensuring that this kind of continuous evolution doesn't create a lot of friction for the overall organization.
"That which we measure, we incentivize towards," is one of her favorite sayings, and Teresa applies it to scoping an overall product roadmap for a company, including what tools, new and old, to use along the way.