Episode 96: Erika Chestnut of Calendly on Breaking the Mold of Quality Assurance

Brian Dawson: Hello. This is DevOps Radio and I'm Brian Dawson. Today, we'll be having a conversation with Erika Chestnut. Erika is the head of quality assurance at Calendly. I'm sure we all know Calendly. I use it every day to manage my calendar, but not only is Erika the head of quality assurance, after having had discussion with her, she is a passionate expert in the subject of quality and quality software and we'll talk a bit about what that means. But hello, Erika, how are you doing today?

Erika Chestnut: Hey, Brian. I'm well. Thank you for that introduction. It's very _____.

Brian Dawson: Oh, well I don' really think – look, I've had a chance to talk to you, research you, and I actually don't believe that introduction does you justice. So, stumbled on my words a bit, but actually to start to, you know, enable our listeners to better understand why I say that doesn't do you justice, maybe we can start with you giving them a brief overview of your role at Calendly, what you do today, but also your background, how did you end up where you're at today?

Erika Chestnut: Sure. So, I'm head of QA. I've been at Calendly for two years. I joined the organization to really help focus in on quality and growing the quality effort within the organization, it's an engineering born organization and so it was rolled in, but the QA team really wanted to have a voice at the table and I love that for them.

So they advocated for themselves and said we need someone here who's helping us, who's helping us grow, who's making sure that we have the things that we need to do our job exceptionally well and the leadership agreed and so I came in and that's how I support them as being oftentimes a bullhorn, you know, a megaphone for them and really empowering them and making sure that they have a seat in all of the conversations to do their job well.

Brian Dawson: Yeah, we – and I'll fork for a minute and then pull back into your background, but I want to raise something that we talked about offline. You said you often act as the bullhorn, the megaphone, the representative of that team to give them a voice. We had talked about how quality or the role of quality within an organization sometimes has aspects of quality engineers feeling like they don't have a voice, they're not heard. Can you talk a bit about – do you agree with that? Do you see that? Can you talk a bit about how that surfaces?

Erika Chestnut: Sure. That's typically my experience. That's typically what I see. You know, we're in a lot of ways, even quality engineers that are developers or have that developer background or some experience are dismissed sometimes. You know, they're – it's like, "It's just QA. You're just testing." Okay.

Brian Dawson: "You don't know. I got this."

Erika Chestnut: Right, "You don't know about architecture. You don't know how to do this thing, how to build this app. You're just testing." And it's not – sometimes it's like, well, sometimes it does come from a point of arrogance and conceit, but other times it's not intentional. It's just kind of this mentality that exists. It's like, I don't expect you to know or understand this conversation.

And then for those that sit in that space of, "I don't expect you to know or understand," when they do – when testers do show up and they are able to engage in the conversation, they're surprised. Right? And that seems to be the case, and that's some of the things that we need to change.

But absolutely there's – that's one of the things that I find that I always have to start with is that I've got to empower my team members. When I'm coming into a new company, when I'm starting a new team, when I'm hiring new team members, there's always this thread, there's always this – it could be a thin layer or a cinder block of you don't know your value when it comes to you as a tester and what you bring to the table and I need to help you.

I have to start there so that they feel empowered enough to speak up into these conversations, but they do more than just show up and be a fly on the wall in the meetings. They say, "No, I have some feedback for you. I have some insight. I have some ideas. My voice is valuable. My ideas are valuable. Let me engage in this conversation."

Brian Dawson: So you hit something. Sorry to interrupt. Now, we know right, there's a dynamic with the developer and a level of arrogance in some cases, condescending views. We also – you talked about having them make the individuals understand their value, but then what about the organization? Right? What do organizations need to do to ensure that that value of the quality organization and quality individuals is recognized and not overridden by some of the dynamics with developers or the engineers.

Erika Chestnut: You're talking about the culture of quality, right, and –

Brian Dawson: Yes.

Erika Chestnut: That's where leadership comes in. That's where like my team was advocating for somebody to come in because they need improvement in the culture of – the quality culture in the organization needed to improve.

And at Calendly, it wasn't that. A lot of great things in place, developers were doing test automation, but there were small things that made it more difficult for testing. And it also made it difficult to move the needle. And so they needed – you know, they needed an advocate. They needed somebody who was going to focus in on that specific area.

So it's about changing or helping with the culture of quality and that discussion, and that is a leadership discussion, getting the right people in those rooms and in those conversations to talk about, do we have enough testers? Are our testers over sourced? Right? Are we sacrificing our ability to test and saying, you know, well, we've got test automation? That's enough. And then who's writing the test automation and what does that strategy look like? How are we thinking about how we're creating quality or how we are assuring quality? How are we thinking about it as an organization. What are the standards we want to have?

So, having, you know, having a voice and having a space at these larger conversations so that you can drive that across a department, across a business is how you cultivate a good quality culture in an organization.

Brian Dawson: So perfect transition, actually. Back to background, now you, while you've spent a number of years working in quality as a quality advocate mentoring people in quality, speaking about it, that's not actually where you started.

Tell me a bit about where you started and then again, how you ended up in the quality sphere in your role that you're in now.

Erika Chestnut: Yeah, that was – it was actually an interesting journey. I started as a help desk analyst for Microsoft. People called in and they would – I was one of the people that – actually it was HP. I'm lying. I started with HP first and then I moved to Microsoft. That was the floor above us. That was the upgrade.

HP, I was supporting, you know, people would call in and say, "My cup holder doesn't work." _____ happened. Really, really happened, I kid you not. That was – I'm dating myself by telling you that, but those were the times when people were like, "My CD player," 'cause we don't even have CD players on computers anymore let alone desktops, but yeah, I started –

Brian Dawson: Wait, I've got to jump in 'cause I think I've heard something like that before, but I don't know if you're being facetious or if people actually thought their CD drive was cup holder?

Erika Chestnut: Yes.

Brian Dawson: Wow.

Erika Chestnut: I mean, that legit happened and then they would put the mouse on the floor and try to step on it like _____.

Brian Dawson: Oh my goodness. Okay. My, how far we've come.

Erika Chestnut: I mean, I was a trainer for a stint as well and I had to show people the difference between a right click and a left click. I am not this old, I am just seasoned.

Brian Dawson: Right, right, right, right. no, but you engaged in important times and I got to think as I stop interrupting you and let you get back to telling the story that that experience, that experience with users calling help desk must have informed your later work.

Erika Chestnut: You know, I've never thought about it, but yes. I used to – I used to have a coloring book, just to kind of like have something to do while I was talking to people, but I always enjoyed talking to all these different people and kind of you know, some of them just ran into the same similar – just the same problem over and over again and so I would always find something to kind of occupy my mind, but there were always interesting people to listen to, you know, different cultures, accents, all kinds of experiences calling in, but I did. I started with help desk and moved up to – I upgraded and moved up to Microsoft Support. And eventually, I moved on to deskside support and around that time, I'm dating myself and it's not good.

Brian Dawson: Oh, don't worry about it. We all know. We all know how young you are, Erika.

Erika Chestnut: But I started, the internet really started to take hold and I would – people would ask me what I did and I would say I work on computers and their response was, "Oh, you build websites? I need a website." And I was like, "No. I said I work on computers. That's completely different words in that sentence.

Brian Dawson: I always loved, "So you say you do computers?"

Erika Chestnut: Oh no.

Brian Dawson: Especially at that time. If you had – if you said you do anything with computers, people would assume that you could fix their hardware. You made – you did everything.

Erika Chestnut: Now, they could have asked me to fix their hardware. I could have – I was like, "Yeah, I do that." Right? Like, that's what I do, but no, they were like, "Oh, you build websites." And so I stepped into – at that point, I said, well clearly there's something here and I need to learn about it. So picked up a Sam's 21 Days, How to HTML in 23 days and learn HTML in 21 days by Sam. I don't even know if they have those books anymore, but I started poring over that and learning about it. My dad had a business and he's like, "I could use a website for this." And so I started tinkering around and building a website for him and built a website for myself. I had blink tags and all kinds of crazy stuff.

Brian Dawson: [Laughs] I'm sorry.

Erika Chestnut: I don't use that use anymore and there was audio [Laughs]. Yes, cringe worthy.

Brian Dawson: That's hilarious. I'm sorry. That was involuntary laughter at the blink tag. That is hilarious.

Erika Chestnut: We all know that that's a bad thing, but it's been replaced with something else now, so..

Erika Chestnut: But yeah, I did that and I wound up getting a job at a startup that needed a front end developer and then while I was there, I was learning classic ASP. This is before .NET.

Brian Dawson: ASP, what is that?

Erika Chestnut: Yeah, Active Server Pages, the one. So I learned that and I just continued on my journey. I got a different job with better opportunities and expected more of my skillsets and I finally wound up at a company called Moxy, which is an advertising agency and I grew there. I spend almost, gosh, almost nine years there growing my career. I moved into leadership. I was leading development squads and at some point, as a leader, I started to talk about quality and be more concerned about the lack of quality in some areas that we were putting out.

And that led to me kind of jumping, fast tracking to that that actually led to me moving into quality because I had had a discussion about bringing in moms to support our test efforts, because we didn't have a quality team and we had quality issues. And I had a friend of mine who was a mom and she was trying to get back, just in the industry and getting back to work and I was talking with the founder of the company and she had this idea, she was like, "What about, you know, it could be called Moxy Moms." And I was like, "That's a great idea." Started to _____ and I realized that we had foundational problems that prevented us from being able to outsource it in any way.

Brian Dawson: Now, if I may, 'cause I don't want that to get lost, before you continue, Moxy Moms, at that time, not to date you, that concept of putting together a program where you engaged and involved new people in the workforce that would traditionally be _____ or in a manner that would traditionally push people out of the workforce. To do that at that time, that's a fantastic idea. I commend you and I commend your founder. Sorry to interrupt. I just didn't want that to go unnoticed. This was 13 years ago. Right?

Erika Chestnut: We didn't do it. We didn't get to do it. Don't get me –

Brian Dawson: Oh, okay. Okay. Okay, sorry.

Erika Chestnut: We didn't get to do it because we had – we had foundational problems. Now, had it been a side project, had I known enough about the industry to say, "Oh, this is a program I can create and market and sell to businesses and help them get up to speed." I didn't have enough of that savviness at the time to – or that background. I had savviness in some ways, but not like – I didn't have that background to understand that at that time and –

Brian Dawson: The power, right, the power of that idea. But okay, so and I interrupted your story because I had misunderstood from earlier, but in seeking to implement that, you're saying it uncovered foundational issues that needed to be fixed first.

Erika Chestnut: Yeah. We couldn't. We didn't have what we needed to share with somebody else, for them to understand what was right or what was wrong. Like, how do you tell me what's not working if you haven't communicated what's right?

Brian Dawson: Right.

Erika Chestnut: What the right links are, what it's supposed to look like, what's the copy supposed to be? I mean, just the basics, you know? And we needed a way to do that and well, first we were talking about a way to do that, but I was like, "We are not doing any _____ what our problem is." And so after those conversations, you know, I had a good relationship with the founder and the owner, excuse me, the heads of the company _____ I was probably one of the earlier people hired, like the 23rd employee.

So I had good relationship with her specifically and she suggested that I open up the quality department and my response was, "Absolutely not." Because I didn't want to be blamed, you know, and I looked at QA as the person that carries the burden of that blame, which is not true because we don't create the problem.

Erika Chestnut: But I said no and there was some – lots of conversation. There were some bribes, a little bit of alcohol. And I eventually said yes and I fell in love with it. Fell in love with the process and organization and strategy. I found my passion in that. That's what I really was latching onto in leadership was the ability to build those things out and look at process and structure and strategy and how to connect people and things and as a part of that, you create quality. You infuse quality. You enable quality. So yeah, that's my thing.

Brian Dawson: Yeah, well no, that's an awesome story. I mean, there's a lot there from the opportunity or the recognition of the – the opportunity you had to have a relationship with the founder, who, by the way, was a female founder, the fact that she recognized the need you arose and then after bribing you or prior to bribing you, entrusted you with the position.

Curious as before we move off of, you know, some of the career discussion, I love that we're able to hit some key points on quality, but before we move off of the career discussion and dig wholly into practices, I did notice that your growth into leadership roles not only in your role as a developer, but then of course now in your role in QA has been exemplary of someone that's striving to have more of an impact, do more. I'm curious to ask, and hate saying this, I'm trying to navigate my way around it, but you're ascension into leadership prior to your current success's leadership has been pretty fast. What is your personal secret as a developer, as a quality expert? What is your secret to your success in working in the technical world?

Erika Chestnut: Oh my goodness.

Brian Dawson: Do you have a mantra that you come to work with every day?

Erika Chestnut: You know what, I've developed mantras on the way. That's a question I can answer. I've developed mantras from every company that I've worked for, I've come away with at least one mantra.

Brian Dawson: Oh, interesting.

Erika Chestnut: My big one is grace and patience, and give it and receive it. That's the like if – my mantras are like my core values, grace and patience is like my top one. You know? And I think when you do that, when you try to give people grace in their interactions or they're following you or you give yourself grace in your own interactions and your values and you're patient on both sides of that conversation that it's going to – it's always going to develop those relationships. It's always going to allow good relationships to develop, which is the only way that you can be successful is when you have relationships and can collaborate with others.

Brian Dawson: Wow, that is grace and patience, I love that and I will probably use it. It fits so much with my view on culture and how you achieve effective teams. You know, I often use the term, you know, "We're all fallible," and we have to be honest. We have to continue to try to improve as individuals, as coworkers, as team members, but when we can be genuine about our fallibilities and patient and understanding of other people's fallibilities, then we're more able to come together to cover those up and deliver as a team and I think that very much aligns with grace and patience. I love that. I love that.

Well, I won't twist your arm and make you share your secrets to career success with me, but it was notable, right, as I was researching and looking into your background, it's pretty impressive and I think pretty welling and I hear some of it as I have a conversation, conversations with you. You're passionate about this.

All right, now that I've made you uncomfortable and kind of stumbled my way through that one, last August you did a webinar on product quality and it touched on how it doesn't start the finish line, but rather it's the entire product life cycle. To that end, well, actually let's stop there. Can you talk to that concept for me? What was the overarching concept of the webinar?

Erika Chestnut: I'm trying to remember. Like where did you dig that up?

Brian Dawson: Well, I know this. Here's where we go. Yeah, we do digging. I have a – do a little digging myself and we have the fantastic Brittany Bell who will tell me where you went to kindergarten. But now, but let's talk about – you hit a point. Without the specific webinar, let me just ask you to expand on that statement, product quality does not start at the finish line, but rather it's the entire produce life cycle.

Erika Chestnut: Yeah, that's so much about where oftentimes QA gets things that are thrown over the well and it's at the very end of the cycle and you touched on it earlier, right, there is – QA is time is sacrificed. Testing time is sacrificed, whatever you want to call it, QA time, testing time is sacrificed, validation time is sacrificed.

Brian Dawson: And I'll restate, you're saying when the schedule gets compressed or something slips, the inclination tends to be to take that time from the validation.

Erika Chestnut: Yep. Yep. And so you can't – because of that, because that naturally happens, whether it's intentional, which it never is. It's never – there's never and intent to say, "QA is not valuable or testing not valuable and therefore we're going to cut that time, "But it is where it gets compressed because it's at the end of the schedule. And so what happens at the end of the schedule? You start to take shortcuts and if testing is left at the end of the schedule, then there you go, product _____ quality is sacrificed. And so it can't start at the end.

You can't have high quality if quality checks are happening only at the end because just by the nature of a timeline, cut corners the closer you get to delivery date, especially if there's no wiggle room to move the date. So you have to move that, you have to move that conversation earlier up, and it costs more. It costs more when it's further down the line. It costs more to change a line of code than it does to change an image, to change a wireframe. It costs too much more and it costs way more once it gets to production, of course.

So we can't – we can't expect that we're producing high quality when we're cutting corners at the end of a life cycle. And that comes up in a number of ways when we didn't include a testing cycle or a validation or appropriate time for that validation when we were talking about how long it was going to take to deliver it. We didn't say it's going to – we didn't include testing as part of the definition of done.

And so you're expecting that done is today, but testing's like, "Well, wait. It's going to take me four days. It's going to take me a week, to weeks to get through this, so we need two weeks more." And if you're not looking at that, you know, again, you're starting to compress, "Oh, I need you to cut that down. Is there anything you can do?"

I mean, if I'm looking at this – if I can cut down in this area and when we can roll with it because not many of our customers are testing, or using in this area of the application so – and those are things we still want to have as part of the conversation because we don't want to test just for the sake of testing. We want to have valuable testing and make sure that we're spending the time appropriately and not having lengthy –

Brian Dawson: Right, but you can't do it by – you can't do it by simply changing the duration column in a project plan. Like, that pressure to be more efficient I hear you saying is good. Right? It's a lot of what has driven our continuous integration, continuous delivery in DevOps discussion, especially as we got to intelligently automate. You know, we can't run full regressions all the time. But it needs to be that deliberate thinking, not in a reactive , "Hey, can you do this in two days instead of five?"

Erika Chestnut: Right, right. It needs to be intentional. We're making this decision. We know we're making this decision up front. We're cutting – we're saying, "Hey, we think we need five days. Do we really? Let's evaluate that. Why do we need five days? Do we need to test this?" We're asking those hard hitting questions to improve our efficiency and to make sure that we are testing the right things and we're having the right conversations about quality.

Brian Dawson: Yeah, fantastic. Yeah, I was having a conversation the other day with my brother who is overseas quality at GoPro and we were talking about – I'm curious to explore this a bit with you – in an organization that has a PMO where the product life cycle is being run by a project manager, we were talking about how important it is as you transition to representing quality across the entire life cycle, as you transition to a quality first organization, one of the stakeholders you need to bring along, 'cause they're going to be in rooms where quality is on at times, is with your project managers, whoever is owning dates and delivery dates. Were we wrong or do you agree with that?

Erika Chestnut: Do I agree – I might need –

Brian Dawson: I'm sorry. Yeah, this is – this will be an edit point 'cause I fumbled on that, but the question was, do – is project – are project managers and project management part of getting an organization to transition to a quality first approach? But what I'll do, Erika, 'cause we'll just edit that out, as I'm actually going to dig in by transitioning into the how can engineering leaders convince teams question. Fair?

Erika Chestnut: Fair.

Brian Dawson: Oh, good. Awesome. So, you know, as we talk, Erika about transitioning to our quality to cover the entire product life cycle, whereas I think a term you used or I use often is a quality first organization, a quality culture, how can engineering leaders convince teams to break the mold and think different about quality, to think more holistically about quality?

Erika Chestnut: We need to talk less about quality engineering or quality assurance as a step in the delivery process, in that validation step, that _____.

Brian Dawson: I'm going to continue to be rude. I'm going to interrupt you there on that 'cause you said quality engineering and then you switched to quality assurance, so before you – what's the difference. I've kind of used them interchangeably?

Erika Chestnut: Sure. So – and I'm sure there are people, you know, that will – there are some that will agree and some that will disagree. Right? I think some people do use it interchangeably. I believe they are distinctly different. Quality assurance is that validation step. It is that silo, is the, "I am checking something here."

And there's some nuance even around that of I'm validating, I'm not looking for you – I shouldn't be looking for your bug. We should able making sure that it's doing what it's doing. You should have been checking for those bugs earlier. Quality engineering is across that delivery, from analysis through maintenance. It is the ability to infuse and empower quality practices or infuse quality practices and empower others and enable others to infuse those quality practices and standards across that delivery life cycle, so we're engineering quality. We're building it into the delivery life cycle, again, from analysis all the way through maintenance as opposed to sitting in that validation stage.

Brian Dawson: Yeah, I love that distinction. Makes perfect sense to me. I think all too often, you know, words matter, not to obsess over words, right, but words matter. I think in my past, I've seen or have used quality engineer to refer to that more technical person that is working in more of a white box manner. But I love that definition. Right? I think that's an apt and the way I viewed it and probably goes back to what we said earlier. It's not about measuring a technical dimension. It is about your view on quality in terms of where you started early in the process, so I love that. Thank you.

So I rudely interrupted. The question, if you remember what you were saying, about how engineering leaders can convince teams to break the mold and think differently.

Erika Chestnut: Yeah, we need to talk about is as quality as a step and talk less about it as a step in the delivery process. So it goes back to how we shift from quality assurance, that mindset, to quality engineering and view it as a construct or conceptual elements, right, that includes testing, that includes that step, that validation step, but instead how do I – how do we engineer quality in design, how do we engineer quality in analysis, in architecture, what are the things that we can do that improve and create opportunities to improve quality?

And it's – it means an extra step. It's an extra step in the conversation and that's why it oftentimes gets overlooked or intentionally overlooked, you know, not even overlooked but stepped aside because it's more work. It's more work to think about how do I engineer my code to ensure that it's testable or that I can test it earlier or taking the step with TDD, do I test first?

Brian Dawson: Yeah. Actually so that gets to, god, a couple of great questions that I want to get into and I know I'm going to run short for time with you, Erika. I think we had realized when we spoke before that we need a three hour episode, but there's a dynamic, so especially as you bring up TDD. I already wanted to ask then how do I frame it, it's to say that, you know, we've moved into this era with test automation, CICD, cloud, faster pipelines, to this era of full stack developers and I've now seen a number of organizations that felt they did not need to have dedicated – a dedicated quality function or people operating in that role.

So sort of the full stack developer. I'm curious, you know, do you feel or with what you've seen, does it work to have a developer both own code and quality and then in the cases where you do have a formal quality function and you have developers, if we're going to shift left, if you will, quality, how does – you know, how can a developer make that relationship more fruitful? So can we have full – yeah, _____ you got the question. Sorry, I was going to summarize.

Erika Chestnut: No, that's okay. So the first question is, you know, do I think that full stack development should cover quality, does it work?

Brian Dawson: Yes. And what I'm going to do, and sorry I'm so clumsy this interview, Erika, what I'm going to do for a second is just parse the questions out and then in a more succinct way, 'cause I stumbled and then give them to you.

Erika Chestnut: All right.

Brian Dawson: So, that does bring me to want to ask, in a world where we've moved to automation, CICD and DevOps and there's just real emphasis on full stack developers, to the point that I've seen organizations that don't build out teams with dedicated people focused on quality but rather expect developers to do it, what are your thoughts on that? Have you seen that and do you feel it's effective or are there some things left on the table?

Erika Chestnut: I've definitely seen it and if a company has it, they'll make it work. If a company doesn't have it, they'll make it work. Right? So, I don't – there are two – there are many trains of thought on this conversation. I have been part of organizations where we had a quality function and they said, "We should be able to handle this," and they have dismissed the quality function. I've been part of organizations where they haven't had it and they said, "We need a quality function because we have our full stack developers. Calendly's one of them. We had a full stack developer, but we need some focus – but we have a few manual testers but they can be doing more." We recognize that and we need somebody to own that conversation and to help us with what that looks like.

So I've been multiple times in both of those seats and that's why I say, if a company – if a company doesn't have it, they come from a space where they started with quality and as they moved into CICD, moved into DevOps world, they were like, "Oh, we should be able to handle this." And they have their struggles. They absolutely have their struggles. But do they make it work? They can. They can. But more often than not, you know another company I work for, they went through that. They said, "I don't think we need our quality function. Our developers can handle it." But they still wound up having to fire automation in _____.

Brian Dawson: Right.

Erika Chestnut: The still had to go down that path because their developers could not maintain both at the speed that they were trying to move. You also have the problem of, you know, when developers handle that, if developers aren't well versed in testing strategy and methodology. They are not looking at how they're building their tests as a framework, as a – they're not looking at the structure of it. They're not looking at how to be strategic with it.

And so you wind up with a bloated test suite, but the potential for that is higher because they're focused on the requirement that they're building and saying, "I need to test this requirement," and they're checking that box. And not all developers are doing this and I'm not saying that it's a bad thing. I'm not trying to bash developers, but there's a different mindset and there's a view, a strategy around our testing that we need to have, whether you're building tests or manually testing. There's a strategy around how you do that that is missed often when you have full stack developers that are owning testing and nobody's looking just primarily at what does our test strategy look like.

Brian Dawson: So right, so even if you do have a full stack team for those organizations that make it work, you need some deliberate and dedicated focus on strategy.

Erika Chestnut: And oversight of it. You know, what does our test suite look like and why does it look this way and what are the things that we are, you know, where are we focusing building our tests on first? Is it unit tests? Are we building more UI tests and why and what is – what are we – what type of tooling are we using and how do we moderate it, how do we make sure that we're not creating redundancy when we're flying through, you know, these PRs, how do we make sure that I'm not writing the same tests that my neighbor is writing or somebody on the other side of this application that's working on something is writing a portion of the same test that I'm writing.

Brian Dawson: So, to that end, right, where you do have – well, in their case I guess this question would apply, I guess I'd ask, how can developers collaborate better to ensure quality across the software development and delivery pipeline?

Erika Chestnut: Is this with or without a testing 'cause I want to answer with the testing.

Brian Dawson: Let's answer 0 well, answer the one you want to answer and then maybe we'll get to the second.

Erika Chestnut: So there should be pairing. Devs pair all the time. There should be pairing with QA. Dev and QA need to pair before development starts. What are we building and how are we going to test it? How are we building it and how are we going to test it? Let's talk through that. What unit tests are you going to write so that I know where I need to – I was talking about this in another talk where this concept of defragmenting and playing Tetris with your test pyramid. And so we're going through that now where I'm saying, "Hey, got our test pyramid. We said this is what we want to include in our test pyramid beyond just the traditional three layers, right, we want to also include component and contract testing."

So now that we know what we believe we need to include in our pyramid, let's talk about the foundational strategy, which is unit testing, and then let's look across the rest of our pyramid and say, "What should we defrag? Are there tests at the UI layer that should be at unit layer? Are there portions of tests that should be broken apart and be put into the unit layer?

Let's defrag that and then let's play Tetris with the rest of our pyramid and plug in the holes, plug in the gaps to create that solid coverage. So, to do that, developers, they should be talking with QA. What does that look like? How do we partner what QA is doing, if they're sitting at the top of that pyramid, with what they're doing if developers are sitting at the bottom and how do we inform each other so that we reduce that redundancy.

We create great coverage, but we do it in a lean way and a way that is less flaky, you know, we're talking about infrastructure as well, when it comes to like _____ a way that is – that models good test strategy, that we know that we're getting the coverage, that we can look at our application, we get the reporting back so we can look at that and say, "We know what we tested. We understand where the failures are. We can quickly respond to them. We don't have a bloated test suite, so we're not extending our build time," all of those things. This is about that whole conversation starts and stops and relationships, right, relationships and working together, that partnership and collaboration with Dev and QA in the conversation.

Brian Dawson: Awesome. And this leads to a question that I've had, how – how can a quality control engineer better survive or better interact in an agile continuous delivery world? Does their role change? We take traditional manual testing and I think about where you talk about the pyramid, there's some of those steps that are going to have to be pushed down to meet cycle. Are there other things that our listeners in quality or our leaders need to think about as quality shifts into this new era?

Brian Dawson: That has been a constant thing that I hear from QA professionals in the organizations and that is that the company just switched to agile and they switched to continuous delivery. Now they expect us to run full test cycles every two weeks or the other thing I'll hear is, "You know what, the development team decided to adopt DevOps. We were never told that we were never brought in." So I'm hearing about this sort of constant tension between the way they've traditionally tested and being caught between a rock and a hard place, like we can't do our job. We haven' been brought to the table to figure out how to do our job at this new pace. And I was just eager to uncover that a bit.

Erika Chestnut: Yeah. You know, I remember when I was at Kabbage, it's a fintech startup – no longer startup, they were bought by Amex _____ bank, but when we shifted to agile, the founders of the company did a great job by investing in things, so we were going to stop work and train everybody. But I remember it being a very big shift for the QA team because we went from we were a team and we rallied and we sat together and we had our culture and relationship and to now you need to break apart and you need to go sit with what are now going to be your squads. You go sit with the developers and the developers had other developers. The QA was the only –

Brian Dawson: Oh, that's interesting.

Erika Chestnut: One in their pod in that area. And while – you know, everybody had – we knew everybody. Right? We were all cool. It was a startup, so we'd been working together, but it was a different energy and mentality and it felt like QA was going into a silo and going into – even in spite of us all getting the same training, you know, and at that time, it wasn't – the conversation wasn't around DevOps. It was just around agile, but that was still a big shift and it was a scary shift because they said we're working in this application, this monolith in fact, and how do we make sure that, you know how do we not step on each other? We're working – we didn't have multiple test environments, right, which is some of that tooling comes into play today. And that's what QA had _____ the answer of that question or that portion of the question is QA has to say, "This is what I need."

Like, don't always start with, "How do I solve it? How do maneuver around it?" It's okay to say, "This does not work for me." It's appropriate to say, "Can we do it differently to help me do my job better?" Don't – we've got to stop accepting that, you know, what's just given to us, which is often the space that QA sits in is this is what's given to me. I'm going to make it work because that's what makes – that's where I get the accolades. That's what accolades, that's what makes me awesome is that I make it work against all odds.

Brian Dawson: Fantastic.

Erika Chestnut: And we can and we should – we should – one of our core values at Calendly, I love Calendly's core values, it's the only company I've ever worked at where we talk about core values the way that we do in day to day conversations, we hold each other accountable for it, but one of our core values is find a way. And we should find a way, but another core value is strive for excellence. So in finding a way, are you also striving for excellence? Is there an opportunity? Do you find a way temporarily, but then do you – in striving for excellence, do you make adjustments in the future? And QA is so often, we just focus on finding a way.

Brian Dawson: Right, which oftentimes goes back to that word earlier, shortcuts or just working within the space you are given as QA versus, you know, on top that strive for excellence, which will cause you to push or strength a bit. Love it. Love it. And I think it would suffice to say that in the interest of time, simply that there's no silver bullet to how quality and developers embrace this new era. There are best practices that you have in some of your webinars and some of your other assets that you have out there, but is it fair to say, there's no silver bullet. It takes work.

Erika Chestnut: Oh yeah. There's no silver bullet in anything. Every company is different. Every product, every business structure, every person that works at a company is different and you bring all of those pieces together and you're going to come up with a different solution and when people come into these conversation and they say, "I have the way."

"No, you don't." You don't because the way that you have was prescribed for a completely different environment.

Erika Chestnut: And there may be some things that you can pull on but, you know, I – I _____ people say like, you know, I've been in interviews where they're asking what, you know, what would you – what would be – what would you prescribe for this? And I'm like, "I don't know yet."

Brian Dawson: I don't know. Right.

Erika Chestnut: "I can't tell you that because I don't have all of the information to understand what's lacking." You know? And so no, there is no silver bullet. You have to work with what you have. But again, those goals, those core values, excuse me, you know, strive for excellence. Find a way. Start with human. That's what leads us and that's what I love about Calendly and our core values is because we're all talking about that same thing and so we're saying, "Does it align with these core values?" And that drives our decisions.

Erika Chestnut: Every company is going to be different. Their driver is going to be different and so you're going to make different decisions. Your silver bullet in that company is going to be different based on what is their driver.

Brian Dawson: Yeah, like we say with DevOps transformations, you can look at frameworks. You can digest information provided by domain experts, but at the end of the day, you need to translate those into what is going to work in your org. I think that's fantastic advice, Erika.

So I want to now shift to one of my favorite parts of these episodes, these conversations and that is what is a dev-oops moment? Again, not DevOps but dev-oops, O-O-P-S, that you've experienced in your career? You know, a moment where you've messed up, you missed something, something happened. It may have been dire consequences, but it's something that you learned from that made you better and can make our audience better?

Erika Chestnut: Oh. You know, I burn those from my memory.

Brian Dawson: Someone else said that.

Erika Chestnut: I really do.

Brian Dawson: That's funny.

Erika Chestnut: I really do. I want to say, it's usually around my lack of long term IC experience in QA 'cause sometimes it feels like I stumble over it sometimes because I went from development leadership into QA leadership and so, where I've had problems is connecting sometimes like what's happening on the ground level, like in the weeds were I'm saying, "But strategy and process." You know?

And I'm like – so, I know I've had hiccups around that and for me, that's also been part of what's contributed to the type of leader I am because I rely on that collaboration and partnership with my team members and I talk very – I talk a lot about like, "Hey, this is my idea. Make Swiss cheese." Right? Like, what – where are the problems? What are we missing?

And if I have to pick one, I think when I was at Kabbage, the CTO, he wanted automation and I said, you know, we need our developers to do that because the application itself was very complex underneath and I said, we need our developers to do it. And I did not push on that because they were like, "No, we can't have our developers do this. You need to hire, you need to outsource. You need to find a way."

And I had somebody on my team that was really strong as an analyst and so I kind of pulled her over into this role where I really had her partnering with QA and building these robust test cases to prepare us to provide it to a vendor and I partnered with some of the engineers in the organization to help me identify a vendor that could come in and build our automation framework. And it was flop. It was – it was – oh, it was a horrible flop in so many ways but it was because they did not have the foundational knowledge. They didn't – our system was very complex underneath the covers and it was changing so quickly. I mean, so quickly.

And it was destined to fail and I didn't – I didn't have enough confidence in what I was saying or enough to back it up to explain why I knew it was not going to work to not have our developers in house or _____ in house building out a framework and building out test automation. And so it didn't. It was a major flop. It was bad.

Brian Dawson: That is – thanks for sharing. And just to repeat them back, I think there were really two things there that were interesting that you covered. You know, one is, most of your challenges have to do with limited experience as a quality IC. And while I thought your answer might have been, "Well, what I need to do is get on the ground and spend a couple of days working at a tactical level, a couple of days a week working at a tactical level." But your answer, which is great advice is, "I need to ask questions and I need to partner, not assume I know, and I need to partner with the individual contributors to take in account their knowledge and what they're seeing at their level. Right?

Erika Chestnut: Yeah, I try not to make decisions in a silo. You know? It doesn't help anybody 'cause I'm not the one who's implementing or benefiting or not from that decision.

Brian Dawson: Well, and I think it's just generally like really valuable leadership guidance. Right? As you move higher and as your leader, you're looking at a wider aperture, the detail of what you see at an execution level is a little bit harder to _____ and you need to involve others. Now the other point, which I thought was really interesting and tied back to this, you know, silver bullet, your other dev-oops was when you didn't push back or you weren't well positioned to make the case for developers working on automation and it almost sounds like there in going with an outside vendor, they were not able to take in account all of the inputs and variables in your organization in order to make it successful, i.e., no silver bullet.

Erika Chestnut: Yeah, yeah. I mean, don't get me wrong, there are definitely products and teams and companies where they have successfully, quite successfully brought in partners, strategic partners to build their automation framework. This was not one of them that it made sense and, you know, later on, they did go down that path. They had to. They had to. But it was all in house.

Brian Dawson: Okay. And it needs to be couched with some in house discovery first, I would tend to say, an in house understanding of what's going on. Well, I'll get to my next question here, Erika. What is a book, podcast, or resource that you absolutely recommend the audience go out, read, follow, watch, consume?

Erika Chestnut: Yeah, my favorite one, I still can't put it down or stop recommending it is Leading Quality by Ronald Cummings-John and Owais Peer. Great book. Not a long read, but tons of solid information on how to increase quality, tying it to a product metric, really talking about quality throughout that delivery life cycle and partnership and collaboration. Tons of great content in there. I highly recommend it for any leader.

Brian Dawson: I have not read that. I'm going to put it on my list and again, be sure, for any leader or anybody in the software development or not just for people that focus on quality day to day?

Erika Chestnut: Correct. Not just quality. I mean, I believe engineering managers, I believe individual contributors, especially those who are looking to get into leadership because it talks about how they can – it will give them insights on how they can leverage their voice and how quality is valuable. It just leans too heavily into, it's not that single validation step and how we should be involved across the conversation and opportunities to do that and ways to do it.

Brian Dawson: Awesome. Well, Erika, I have really, really enjoyed the conversation with you. While it maybe sounds like I say this all the time, I really do mean it when I say I wish we had more time together. But before we let you go get back to, as we were joking earlier, dealing with little fires, do you have any final thoughts for our listeners?

Erika Chestnut: Black Lives Matter. Quality engineering is an excellent career choice and if you're not using Calendly, what are you waiting for?

Brian Dawson: That was the most creative set of final thoughts I've had anywhere. Is there anything that you're doing that's coming up soon or any source of information that you have that our listeners can go look at after this episode?

Erika Chestnut: Sure. You can check me out on my website and feel free to schedule some time to connect if you'd like. I'm at Erikachestnut.com. That's Erika with a K, E-R-I-K-A Chestnut.com. I'll be speaking in a few places over the next few weeks, Global Quality Summit, Cobaton and STAREAST coming up in April as well.

Brian Dawson: Wow, you'll be really, really busy. Well I'll look forward to catching some of that and then hopefully I'll run across you. I can't say enough Erika how much I enjoyed the conversation. I know we just skirted the surface here, so I do look forward to seeing some more of your work and hopefully getting an opportunity to talk about quality with you again sometime in the future.

Erika Chestnut: Indeed, thank you so much. It was an honor to chat with you. I really enjoyed the conversation.

Brian Dawson: Likewise. So that concludes another episode of DevOps Radio. Thank you for listening. We'll see you next time.