Episode 79: React Native App Development for Sneakerheads at Stadium Goods

In episode 79 of DevOps Radio, Nick Koutrelakos of Stadium Goods speaks with Host Brian Dawson about react native app development and how the organization creates meaningful digital experiences for sneakerheads.

Brian Dawson: Hello. This is Brian Dawson with DevOps Radio. Today, we have the pleasure of interviewing Nick Koutrelakos, the Lead Reactive Native Mobile Developer at Stadium Goods. Hello, Nick. How are you doing?

Nick Koutrelakos: Hey, good morning, Brian. Doing well. Thanks for having me.

Brian Dawson: So, you're sheltering at home, right?

Nick Koutrelakos: I am sheltering, this is week four or five, I believe, in Los Angeles, yes.

Brian Dawson: Oh, that’s right. We were talking Maryland, so in my mind, I was thinking you were sheltering in Maryland, but you're in Los Angeles, the order came down early there. I know for a lot of us, I assume you, like a lot of us in tech, it’s not a huge change to spend time at home in front of our computer, at least not initially.

Nick Koutrelakos: For me, no. My day to day work life has been largely the same. It’s more about the externalities of my coworkers, my family and my friends around here, that kinda thing.

Brian Dawson: Yeah, yeah. And I unfortunately, at this stage—don’t tell this to the listeners—don’t have much of a life, so it’s not that I get out of the house much anyway. And I was telling all of my friends that, and my wife who’s here, that we're really adjusting, that it’s not much of a change for me. But now, as I go into roughly about week five, it’s starting to feel a bit different. It’s starting to feel a bit different.

Well, I'm glad to have you. Thank you for taking the time during these unique times to speak to us on the subject of DevOps and software development and delivery. 

To get us started, Nick, can you give our listeners a brief overview of what your role is at Stadium Goods and what your background in software development is?

Nick Koutrelakos: Sure. So, my role at Stadium Goods is what we call a Tech Lead, basically, leading the Mobile Software Engineering team, which consists of me, and we have three other front end developers and a couple back end developers, and largely doing agile development for our mobile location which, as you mentioned, is built in React Native, so we have an Android and an iOS platform.

So, I've been with Stadium Goods for a year and a half now, and the first 9 or 10 months, it was just me on the front end of Mobile Development. And so, thankfully, we've ramped that up to a total of four people over the past six or seven months.

In terms of my career background, I've been doing predominantly React Native since the beginning of 2017. I was actually first introduced to it via podcast in late 2016. And before that, I was a web developer and I was very familiar and professionally developing in React. And when I heard it, I was like, this is—this is too good to be true. I've always wanted to do mobile development, where do I start? I know web pretty well, but it’s just a whole different ecosystem. And through that single podcast, it kind of catalyzed the rest.

Brian Dawson: What has now kind of become a career trajectory for you.

Nick Koutrelakos: Right. Three months later, it was about three months exactly after that podcast was when I first started my, I started my first professional production level React Native application for an agency.

Brian Dawson: Alright, so, you embodied agility and upskilling. You gotta tell us, what was the podcast?

Nick Koutrelakos: I'm pretty sure it was JavaScript Jabber.

Brian Dawson: Okay, okay. See, the power of podcasts, DevOps Radio listeners.

So, I'm gonna want to dig into, in a moment, exactly what is React Native, but before we do, can you tell me a bit more about your journey to being a web developer, your experience as a web developer, as that being the face that precluded you joining Stadium Goods as a React Native developer?

Nick Koutrelakos: Sure. So, I didn't have an educational background in development. It’s something I kind of picked up as a kid and then it fell off during university. And I picked it up a year and a half or so into post-grad or post graduating with my Bachelor’s in Business. And I was working in D.C. and just realizing the need for a lot of automation, the need for just added technology in a lot of contexts that I was faced. And so, that got me back into programming. I started doing just some, picking up drops here and there on sites like Upwork, refactoring—moving things from jQuery. A lot of the clients wanted, you know, we have this stuff built in jQuery and React. React is the next big thing—this is, you know, 2015, 2016. React is the next big thing. We went to some of the smaller companies, especially, went to the rewrite or when it just, little components switched.

And so, that’s where I started. I took—that’s where I restarted. I took strongly to React, because it was fairly—I thought it was fairly simple to pick up, and it had a very strong community from the early goings, which was beneficial to me, because it was free learning everywhere, everyone had great advice. Being led by Facebook was a huge pro to me.

Brian Dawson: It just got you going?

Nick Koutrelakos: That got me going, yeah, to investigate. There’s all kinds of stories I could tell in there, but I did—my question I always asked myself is, once I got more confident and more familiar with this tooling, especially around, you know, it’s all JavaScript based. My question I ask—what do I actually like to do? What do I really—what part of my day to day do I like when I'm programming, do I like the most or the least? And that’s where the mobile side of things, I really liked the layout. It was much easier to manage. Even doing web for mobile was much easier to manage than building for all kinds of dimensions and device types.

Brian Dawson: Right. Huh, awesome, awesome. And for the audience that doesn’t have access to your bio, Nick has a Bachelor’s in Supply Chain Management—excuse me, I'll restate that. So, Nick has a Bachelor’s degree in Supply Chain Manage—Supply Chain Management. [Laughter] One more time.

Nick has a Bachelor’s degree in Supply Chain Management, which is very much the way we look at software delivery pipelines today, and I'm curious to ask, I don't know if they're there. Because if you look at the delivery process at Stadium Goods, are there any parallels with the studies that you did in Supply Chain Management?

Nick Koutrelakos: Oh, absolutely. Having 18 months, I know a lot of experience with Stadium Goods, I know a lot more about our whole—our whole process. And it was, for those that aren’t familiar, it was a startup. We're only about four years old. And so, being a startup working at that pace, a lot of things initially were piecemealed together. And, from our warehousing to intake to fulfillment to our whole tech stack, it’s incredible to see where, when I joined, we were at the three year period, now, we're four and a half, where our supply chain and fulfillment and all the essential employees that we have working there, where they've come, how our processes have changed.

But there are, yeah, I mean, I could—many case studies that I studied at Maryland came to mind in my first couple months while I was building this app and becoming more familiar with, you know, the where and the how of how we get our products, which are predominantly shoes, into customers’ hands.

Brian Dawson: So, there’s an interesting thing there, because I was pointing down a slightly different path, right, and saying, “How does managing changes from concept out to device in your particular case mirror the supply chain?” But That is a real topic of focus for mine in the industry lately that I appreciate and it is the power of being able to understand the business within which you work and how that enables you to deliver better software, better capabilities to better support the business.

Would you say that that understanding of the business drive by your undergrad studies helped you be a better developer for Stadium Goods?

Nick Koutrelakos: Absolutely. I genuinely do. Because, I mean, I've worked with people who have more—you know, they have CS degrees, they haven’t ever been exposed to business material about, you know…I also had studied Marketing as well and I'm familiar and I actually like that aspect as well. But if you're never exposed to something like the supply chain process, like a marketing plan to drive more people and then how that affects the logistics of your business, you know, you don’t know until you know.

Nick Koutrelakos: And so, I can, without a doubt, I mean, we have new features, as we build our software, as we build the app which touches many different points across our both tech stack and our personnel stack, if you will, not knowing how something—like, for example, our Product does intake.

Nick Koutrelakos: Not knowing that part would lead to gaps in our scrum process of feature development, because just some certain questions just wouldn’t be thought to be asked.

Brian Dawson: Okay. Right, right, right, right. And I’d like to say—so, a request would come in, it would simply be words on a screen in Jira without sort of life or context to them. And it sounds like with you understanding the intake process, you're able to lend yourself and your team the context around what’s actually written, and it sounds like that results in a more complete deliverable and a more complete planning process, and then ultimately, a more complete deliverable.

Nick Koutrelakos: Exactly, exactly. There are holes that we discover as many companies do in that process that we weren’t previously aware of, either on the mobile team or something that we weren’t communicating with fulfillment, with customer service. And as—and on our team, we also have a Product Owner—as she and I and our other developers become more aware of this, the business context in which the feature resides, I've seen firsthand other developers asking the questions that, you know, three months ago, they just weren’t, they weren’t asking. And they're more product focused.

Brian Dawson: I love it.

Nick Koutrelakos: They're more logistically focused—and yeah, it’s great to see. I mean, it helps everybody.

Nick Koutrelakos: Very, very high level of knowledge sharing in there

Brian Dawson: So, I think you're gaining some experience that, for me personally, it took years for me to gain. And I talk a lot about—especially in Silicon Valley where I started working, but really, it’s prevalent throughout kinda this wall or silo between engineering, sales, and marketing, right? And I for a long time just felt like, “Hey, look, I can code. All I need to be able to do is to know how to code and I live in my domain and you guys are, you know, you guys should be glad we're here creating awesome software for you to market and sell.”

But, you know, I later in my career learned I think what you're getting a chance to learn early on, and that’s that—no, you know what, we're better together. It is really important that we understand the business, and the business understands software development and delivery and when the two work together, you could drive better outcomes—i.e., we're not just here to code cool things, create cool stuff, commit that, and go home. At the end of the day, we're all working within the context of a business, and we really bring value when we can help bring value to the business through the software we create. That’s kinda my take, right? So, I think it’s cool, actually, that we stumbled upon that and that that’s something that you're experiencing now and sharing with our listeners.

I’d like to go back to the fundamentals real quick, because I got excited, we got right in and started talking. Can you start by telling us a bit about who Stadium Goods is and what they do, for our listeners that are not familiar with Stadium Goods?

Nick Koutrelakos: Sure. And when I first came to Stadium Goods, I was not familiar, either. So, yeah, so Stadium Goods, we're a leader in footwear—fashion and footwear. And so, myself, I'm not a fashionista, I'm not a footwear enthusiast—you know, I do like comfortable shoes, but like any, you know, you have trading cards, you have antiques. There’s a collectible aspect to sneakers.

Nick Koutrelakos: And so, we cater toward—

Brian Dawson: Sneakerheads as they're often called, right.

Nick Koutrelakos: Sneakerheads, exactly. And so, we cater towards—more towards that market and, you know, we have the latest drops, as we call them where, you know a finite amount of a specific make and model of a sneaker are dropped on a certain day. We have rare sneakers. When I first started developing, I thought there were some issues in the data, because some of our sneakers were coming in with prices of $14,000.00, $23,000.00.

Nick Koutrelakos: And then I started reading, because I wanted to familiarize myself with this business and that people were—you know, that was the thing. And so, that’s what we do. You know, people come to us, it could be a mom and pop store or someone who is, you know, they own their own business where they go around and look for these sneakers. It could be, other companies are more, have a more established process. They come to us, and say, “Here’s what we have” and we provide the platform on which to, you know, they—

Brian Dawson: For them to market and sell or distribute.

Nick Koutrelakos: Exactly. And so, we also do some—you know, where we have fashion products, like a lot of streetwear. Everything is streetwear related, so hoodies, hats, socks, keychains. But if you browse our catalogue, it’s predominantly, predominantly sneakers.

Brian Dawson: Okay. Yeah, very, very interesting. And I happened to have noticed some Travis Scott Air Force Ones low tops that if you get a chance to get your hands on, you know, DevOps Radio would appreciate it. 

Brian Dawson: You know what we'll have to get you is, the CEO of CloudBees, Sacha, has some prized CloudBees Air Force Ones. We'll have to get some of those going through Stadium Goods.

But okay, cool, thank you for sharing that. Now, also, to establish a baseline, what is React Native? I think most of our listeners, if you've lived in the technical space, if you've participated in software development, really in any aspect, it has to have surfaced sort of into your purview and understanding of the key technology. React Native is a bit newer. I'm not sure that everybody knows what it is. Can you tell us about it?

Nick Koutrelakos: Yeah, so, it’s another—like you mentioned, it’s a take on React, so React…I'll call it React Web. So, React Native is a project also backed by Facebook and developed in house. They have, last I've checked, maybe 20 or so engineers responsible for maintenance and ongoing, but—ongoing development. But it allows…it follows the principle of React where, write once, deploy everywhere.

Nick Koutrelakos: You know, because, for web developers, you have to, there’s all kinds of browser compatibilities and versioning and there’s all—you know, screen sizes and everything. And React kinda was one of the frameworks to set the standards for, “We'll handle that. Just write the code.”

Nick Koutrelakos: And so, React Native takes that to mobile such that you write code that is very similar—it looks, the patterns are the same as React, it’s the React way. You know, you have the component based, component based structure for your app. Some are stateful, some aren’t. It’s the React way. But, at the end of the day, your JavaScript will compile to native, actual native code. So, in Android, it’s Java; on iOS, it’s Objective-C and Swift.

Nick Koutrelakos: And so, it creates—there’s a bridge between the JavaScript and the native layer in that most of your app is, once it compiles, it is actually native. Then, on the JavaScript thread, which is an asynchronous thread between your app and the native layer.

Nick Koutrelakos: That’s where your business logic predominantly lies.

Brian Dawson: Okay. Yeah, and so, you know, obviously, there’s a natural, I think, association with React and front end, right, React and UI implementations [Audio skip] as you talk about business logic, can you help me understand, does React Native have the constructs to enable you to build cross platform native application, soup to nuts for want of a better word; i.e., can you deal with business logic handled data, the tasks beneath the surface of just the UI?

Nick Koutrelakos: From what I understand of your question, yes, that’s—from soup to nuts, Stadium Goods is a great example of that. You're able to, just as you would in your web app, control, maybe have a local data layer. You're fetching data from, you know, mobile APIs and consuming them in some way. And yeah, so, in the JavaScript, that’s where all of that happens. And the UI itself is what predominantly gets—you know, it consumes that data.

Nick Koutrelakos: And then rather than—there are many different hybrid approaches to mobile app development, and some of them are more performant than others. And so, I think that React Native, it provides for great performance with the ability for anyone who knows React to—

Brian Dawson: Awesome, awesome. Yeah, I marvel personally, again, to continue to date myself as an Assembly, C, C++, and then ultimately Java programmer, I still—you know, and then working with JavaScript in the early days of JavaScript when its application and use was very limited, it’s insane to see how widely it is used, how, you know, widely it is spread in terms of application tiers, for use in terms of application tiers. As a matter of fact, to my right is my LG TV running on webOS, which was, you know, purchased from Palm and, you know, as OS built on JavaScript, right? And actually, ironically, in a reverse turn, originally targeted for mobile devices.

So, do you guys practice, or does your team practice agile at Stadium Goods?

Nick Koutrelakos: We do, yes.

Brian Dawson: Okay, okay. And how long have you practiced agile, from the start?

Nick Koutrelakos: That answer really depends. It depends—we have what we call squads. So, we have the mobile squad, we have the dot com, the website squad, we have a DevOps squad, we have fulfillment and customer service. So, these are all different squads. And so, on the engineering side, I know that mobile was, we had many fewer developers in comparison with dot com.

Brian Dawson: Oh, gosh.

Nick Koutrelakos: So, it’s been, for mobile, it’s been a few months. You know, as we've had more personnel join the team, that is when agile in particular—

Brian Dawson: Okay, for that particular—agile kicked in for that squad. So, it kind of depended on the maturity of a given squad or the ability for a given squad to benefit from those practices.

Nick Koutrelakos: Correct.

Brian Dawson: Okay, and that’s—so you guys, your squad, are you guys adopting some version of the Spotify model? Are you familiar with the Spotify model?

Nick Koutrelakos: I'm familiar with it. I’d say it’s very loosely based.

Brian Dawson: Okay. Yeah. We use the term squads at CloudBees, where I work. So, I am curious, so I'm gonna assume you've implemented agile. We've talked a bit about automation, you know, CI, CD in pursuit of DevOps. And I was curious to ask you, what are some of the challenges and/or benefits of those practices that are unique to delivering software in a mobile environment?

Nick Koutrelakos: So, it’s a tricky question in React Native, because it is so new. You know, at this point, it’s been around, I think it was open source in 2015 or March, 2016.

Nick Koutrelakos: So, I mean, it’s in its infancy if you compare it to other platforms. But—so, it’s a challenge that you have tools, for example, end to end testing, if I can use that example. You have more established players like Appium, and then you have other tools that are even newer than React Native, because they were built React Native first.

Nick Koutrelakos: So, another, one of those examples is called Detox. And so, it’s a challenge—in React Native, there are kind of two different silos. One is like, you have custom stuff, you know, you need to use the camera, you need to use Apple Pay, you need to use all of these common bits of the native, on the native side.

Nick Koutrelakos: And so, that is—that’s one silo. And the other one is, well, you need to use the SDKs or components of third party libraries or just some part of the native side of things that isn’t exposed yet, doesn’t have a formal library or integration with React Native.

And so, going back to the end to end testing example, this tool, Detox, which was, again, written in React Native first, it handles the one side really well. Everything that’s built in that one might, that one might use—

Brian Dawson: Into the standard APIs, right.

Nick Koutrelakos: Standard, right. And so you can write different end to end tests, whether it be the on boarding experience is as it should be, the home page, everything loads properly, a user can check out properly, a user can add items to cart, add and remove and all of that, you know, we can automate that.

But once you get to the more, once you get to adding more customization in the context of React Native, then you run into more challenging obstacles, which, they're not, I wouldn’t say they're new in the broad scheme of engineering, but they're new in some way, shape, or form, to React Native mobile engineering.

Brian Dawson: Okay, okay. So, it’s a little bit harder today—so, going back, implementing robust continuous integration, ultimately continuous delivery has, you know, a high level of code coverage or at least intelligent code coverage in terms of your tests, which can then be automated. And I hear you saying, right now, with the tools available, it’s fairly easy to do with tools like Detox or Appium. It’s fairly easy to do it with the standard aspects of React Native, but as soon as you start to push React Native a little bit further, you don’t necessarily have the hooks, you can say, to instrumentate the test against those custom implementations or native implementations, right?

Nick Koutrelakos: Exactly, and I think you said it best when you said, well, it becomes more of intelligent code coverage. Because that, once you do have, you know, you're like, “I can do workarounds for this and this, these custom things,” but you always have to—

Brian Dawson: Hand roll that stuff.

Nick Koutrelakos: Right. And intelligently decide, is it worth it, how can we—you know, does the manual effort here, what are the tradeoffs and just be strategic about it, given all the constraints, you know, expectations are set and—

Brian Dawson: No, I think that’s an important sort of serendipitous discovery there, right? It’s called out as a shortcoming, but you know, roughly used, a common adage, “When you have a hammer, everything looks like a nail,” right?

So, one of the things that are a constant challenge in more mature ecosystems is, people just take a strict code coverage metric as a threshold they need to hit, right? We need to have 80 percent code coverage. People quickly build out tests for everything without really understanding if they're testing the critical points. Eventually, you find (1) the more tests you create, it’s the more tests you have to maintain, right? But (2) your cycles start to get sometimes, as the app grows, your cycles get prohibitively longer.

So, again, you know, I think, speaking to another thing, you may be in a good position, you know, where you can turn a positive into a negative, right? And that’s—look, I have to understand my code. I have to understand the tests that we're writing. I have to engage with automation intelligently, not just—

Nick Koutrelakos: Right, and going back to what we spoke about, you know, 15 minutes ago, understanding the product. It all comes back to that, and the value that’s being add.

Brian Dawson: Love it.

Nick Koutrelakos: Yeah. It all comes full circle.

Brian Dawson: See? Given time to talk, you know, people—I'm sure, at times, people think, “Where the heck is he going?” We're going somewhere. [Laughter] 

So, let’s see, we covered your background, we covered Stadium Goods, and that’s a really interesting business case, actually. We could almost do a whole episode on that. We talk about React and the JavaScript ecosystem in your DevOps and sort of implementation.

Now, I think what’s interesting in all of this is, Stadium Goods, React Native, the tool you use to bring the business to your consumer through a mobile device, all of those point very much towards user oriented implementations, right?

Nick Koutrelakos: They do.

Brian Dawson: And I'm just curious, what is your take on the importance of user experience, you know, specifically in e-commerce and mobile, but then also in software development today? Is it important, is it not important—how important?

Nick Koutrelakos: So, I'm fairly biased, because I mean, one, I grew up tinkering with—you know, like, Craiglist was the standard at one point in my life for, like, “This is how a website should be.” Craigslist obviously hasn’t changed much, that’s kind of their spiel.

Nick Koutrelakos: But everything has. And so, and then being, you know, in school studying marketing, trying to understand the marketing cycle, how do we get you to use our product—my biggest takeaway was, you can have the best product in the world, but if it’s not user friendly, if nothing is drawing you to using it, it really doesn’t matter. You could have the best tech stack, the leanest, the fastest, the most performant, but you know, if that UX isn’t there, and you don’t understand the opinions around different implementations of the UX from an actual big data user’s perspective, then you're losing so much.

Brian Dawson: Yeah, yeah. I think that’s maybe, you know, there was a while where just, you know, being able to deliver function, right, through software development as the tools, languages, and practices matured, that was sufficient, it feels to me like we're in an age where form—function and form are required, right? And form is absolutely critical. 

I think, I'm curious on your comments on this, right, you being in mobile, in particular, I think mobile, along with open source, which you're using with React Native, as well as SaaS have driven a consumption model that I call a delete or replace consumption model—i.e., and it places a burden on us as software developers to engage the user, right? So, if I'm using a mobile app and, you know, maybe my first interaction with Stadium Goods is, I heard about it, downloaded the app on my iPhone, and now I'm using that, and if I had a bad user experience, for a lot of similar software, all it really takes for me to get rid of you is just hold down on the icon, delete, search the app store and replace, right?

Nick Koutrelakos: Yeah.

Brian Dawson: So, my take is—and you live it day to day, I observe it from afar—that has a considerable impact on how you approach development and the requirement to approach it in consideration of the user. Do you agree? Or do you have thoughts on—

Nick Koutrelakos: I 100 percent agree. And as, you know, when you're developing on one product, as I've read a lot of blog posts for similar small mobile teams for applications that reach a large population, you're constantly looking over your own UI, you're booting the app from scratch or you're going through all of these—because in my free time, I could be laying in bed and I'm like, “Is this good? Does this really work how,” and I'm trying to put myself in that position. But like you said, [Audio skip] both in this app and other apps I've developed, I've come to the conclusion in certain sessions where I'm like, “I would probably delete the app right now.”

Nick Koutrelakos: “This needs to be fixed,” jot it down, jot it down, bring the, what needs to be fixed and the reasoning to Product and certain decision makers to say, “Hey, can we prioritize this? This is not the best we can do, and this is why it’s a detriment to our brand, our everything.”

Brian Dawson: That is—I was just gonna say, not only be able to empathize with the user, evaluate the implementation from the eyes of the user, which you sound committed, while you're laying in bed. But then get up the next morning and, as we talked about earlier, now being able to communicate to the stakeholders why it matters to the business, right?

Nick Koutrelakos: Right.

Brian Dawson: Versus just, you know, that is an important skill set to develop. So, do you use any constructs like experimentation and/or feature flagging to help identify into the user experience?

Nick Koutrelakos: Oh, yeah. This is where, you know, Rollout CloudBees comes into play. We do—we do simple feature flags, we do multivariate flags, you know, we do configuration values, you know, which copy, which text, how does it compare in certain places in the app. And so, the Rollout tool has been, you know, is very lightweight. It had an official React Native SDK and those were like checkbox, checkbox. And I think most importantly, it’s easy for Product to use and it’s easy for Development to use. Because there are, as one can imagine, miscommunication there and any friction there really gets amplified in terms of how it drags the development process. Like, here’s what we want to develop, here’s how we want to measure, here are the tests we want to run. But if it’s not usable by both parties, then it’s-

Brian Dawson: Now, just to make sure I'm tracking, by “both parties,” you mean both the developer that created the code and the product owners or other stakeholders?

Nick Koutrelakos: Correct, yeah. Any stakeholders, whether it be Product or, you know, Head of Marketing or our data analysts. If they're not able to really understand how slice and dice experiments compare, then…

Brian Dawson: Right, right. Well, I think what I love is—as a matter of fact, I've been having a lot of discussions around feature flagging. I was actually going back and reading a Martin Fowler article, don’t know if you've seen it, from 2016. Listeners, I think it’s a great foundational yet deep write-up on feature flags, the pros and cons. But one of the things that he calls out—right, well, so, if you've been developing software even, you know, C, C++, I come from games for a long time, you've used Pragmas, you've used conditionals to cut out chunks of code while you test and kind of toggle them. 

Now, as we started to move into applying those to web applications and controlling delivery of features in a SaaS environment, it’s great to be able to do some really powerful things with those at a code level, right? But what Martin Fowler makes a point of is, that quickly becomes unmanaged, that quickly becomes tech debt and complexity that can get out of control, but also not necessarily achieve full value. So, what I'm loving and what I'm seeing and what I hear you saying with this current generation of feature flag management solutions is—yes, the developer can code it, the developer can identify experiments that wanna be run, but having a dashboard or a UI to manage those connects the developer to the business, both working together to better serve the user.

Nick Koutrelakos: Exactly.

Nick Koutrelakos: I mean, because we have both, you know, we wanna run this experiment or we want to wait until a certain date to make this code live. But we also, as you mentioned, internally, we have our own epics and stories and bug fixes that we're working on, and they are also hidden by maybe not cloud connected feature flags, but in the code base, you know, we have feature flags that, you know, there have been instances of them growing to a rather unmanageable state. But they're very clear, it becomes a very clear management, a system of management for, you know, cross merger quests and cross commits, all of that, the development life cycle. It is—I mean, a lot of big companies use it and for us, it’s been very helpful.

Brian Dawson: Have you guys ever had to use a kill switch, flip a kill switch?

Nick Koutrelakos: Once.

Brian Dawson: Once? Okay, so, this may be a good time. It sounds like that may be a rough area to track into, so we won’t. But—

Nick Koutrelakos: Well, it wasn’t necessarily a kill switch, it was more, it had its own context, so it wasn’t drastically, the implications weren’t drastic.

Nick Koutrelakos: But we were happy that we were able to turn it off.

Brian Dawson: Okay, yeah. I was reading a story recently—this is something else to look into, if you're not familiar with it, Nick, look it up, otherwise, listeners, you do, about a trading company that had about $350,000,000.00 in assets under control. They had some code in feature flags, but they couldn’t find the kill switch to flip. So, in a day, I think roughly it was, that algorithm driven by that code resulted in $400,000,000.00 in losses in 24 hours. So, this company with $350,000,000.00 in assets lost $400,000,000.00 due to inability to find the offending code and flip the kill switch on it.

So, it sounds like it wasn’t that drastic for Stadium Goods, right?

Nick Koutrelakos: Nowhere near, nowhere near.

Brian Dawson: Okay, but this does set up for something I love to ask. We like to talk about DevOops, right? It’s not DevOps, D-E-V-O-P-S, but rather D-E-V-O-O-O-P-S. Oops! Can you tell us about a DevOops moment or a software development challenge that you faced in your career and share with us what you learned from it, how you overcame it and learned from it?

Nick Koutrelakos: Sure. So, one thing that comes to mind is that, at one company I was at, we had campaigns to e-mailers, you know, push notifications. So, this happened to be during one of those campaigns, we have, our DevOps team had practices in place to auto scale certain parts of our back end architecture to adjust to the increased demand, you know, those spikes.

And so, part of that auto scaling implementation didn't, it didn't work as expected. And, you know, I think a table or something got flooded, and so we pretty much, the decision was made—and this was on a weekend. This was,  you know—

Brian Dawson: It’s always on a weekend. It always is. 

Nick Koutrelakos: It’s always on a weekend. And it was early in the morning for the responsible—

Brian Dawson: Parties, whatever geography they were in.

Nick Koutrelakos: Exactly. And so, a decision was made—well, let’s pretty much hit the reset button on this. So, it worked, things were great. But then I think I remember coming into on Monday and realizing that the impact of that was, the sessions, the sessions for users were actually integrated or affected by that reset.

Nick Koutrelakos: And so, we quickly, you know, had to, it was a very small team—

Brian Dawson: So, you wiped every active session, effectively? Is that what happened?

Nick Koutrelakos: Right, but there wasn’t a fallback on the app client on the front end code to account for something like this happening.

Nick Koutrelakos: And so, we had—luckily, it was a small team, and so we were able to iterate very quickly and figure out, you know, a hotfix for it. But, you know, as we were coding this and deploying this hotfix, we could see in our bug tracking tools, the users that were affected rising and—you know, it also affected, for a short time but an impactful time, other business decisions like sending out new campaigns and increasing traffic. And we kind of had to readjust. We had to communicate it to stakeholders, adjust, and you know, keep everyone in this loop until it was...until we could get that hotfix deployed.

Brian Dawson: I can imagine. It becomes the situation room in that type of thing, right? 

Nick Koutrelakos: Right.

Brian Dawson: You're reporting to the CMO, the CEO, “How are you coming along with it?” So, what was the lesson learned? What would you do to avoid that next time?

Nick Koutrelakos: The lesson learned there was to have more documentation and more communication from across stacks. So, in this case, it was, if something was on auto scale properly in this specific context or in other contexts, who’s responsible, what do we do, and who needs to communicate with who? Which teams need to communicate before the arrived upon decision is actually, the green button is hit.

Nick Koutrelakos: It’s very similar. It’s actually very similar to what’s going on now with this quarantine and confusion on setting and people not being on the same page. So, it might be a rather cliché lesson learned, but it had more specific implications in the context of the teams and how.

Brian Dawson: Right, and you have multiple—even in what is a smaller company in terms of comparison to, say, a Fortune 50 company, you still have multiple different squads and stakeholders that need to communicate, and it sounds like what lacked on the analogy of where we are now is a playbook that everybody agreed to execute against in this type of situation. Is that a fair summary?

Nick Koutrelakos: I think so, yeah. Yeah.

Brian Dawson: Yeah. I think, you know, my—what I take from that is something that I'm also kind of learning and running into is, when we talk about DevOps automation, collaboration, people over process, I think somewhere along the line, we lost sight of the fact that it doesn’t mean no process, right? Process is still important. Process planning is important. What we need to do is get rid of waste, right, useless wait states that are done just for the sake of doing. And ideally, you know, there’s a term called ECR, enterprise change request process, right, for dealing with—I'm sorry, not enterprise, emergency…there’s a thing called ECR, emergency change request process that you would follow for things like these.

I tend to feel that, ideally, your change process is fast enough to be your emergency change process, but that doesn’t mean that when you have to make an emergency change that you shouldn’t have a playbook as to how people communicate and how we come together to solve it quickly.

Well, thanks. I appreciate you sharing that DevOops moment. I’d also like to ask my one final question for you is, for our listeners, especially, that are interested in mobile development or better understanding the business or specifically React Native, can you think of a book, a podcast, or another resource that they would find value in checking out?

Nick Koutrelakos: Yeah, so, whether you've never touched React Native or you're familiar or even myself, having done it for four plus years, [Audio skip] go on Udemy. Udemy is an online educational resource platform. His name is Stephen Grider, and I have no affiliation with him, but I know that over four years ago, that’s where I started. And the concepts that he instilled, while not perfect, because nothing is perfect, they still stick with me today. They formed a foundational knowledge that was just, it was a great, it’s a great base on which to start. And he’s kinda, you know, the courses are kinda beginner, advanced, in the weeds courses. So, there’s all kinds of content, there.

And then I mentioned, I still listen to it occasionally, the other podcast, I did a shameless plug there for JavaScript Jabber. It’s where I first hear about React Native, and those topics, I find myself on a Saturday or Sunday taking something I've heard there and going down a rabbit hole.

Brian Dawson: But ultimately, it sounds like from what I've learned from you on this call, though, hearing about it on a Saturday, learning it and possibly putting it into practice by Monday, Tuesday, or Wednesday.

Nick Koutrelakos: Right, right. I mean, that’s how JavaScript, that’s how the community is kind of driven is a quick iteration, for better or for worse. You know, you miss a week, I go on vacation for 10 days and all of a sudden, I come back and I know nothing.

Brian Dawson: It’s all changed. That's funny. Alright. Well, thanks. I appreciate you spending this time with us and actually, I think you shared a lot of really valuable nuggets for myself and our listeners to go home and think about. As we get ready to wind up here, do you have any final thoughts, guidance to share with our listeners?

Nick Koutrelakos: Just one final thing, and it relates to the last topic that we were speaking about, you know, you were talking about ECRs, and we, another—to that point, there’s something called ADRs, which is architecture decision record. So, the concept there is that it’s like a piece of documentation that allows you, as a team, to document and capture why important architectural decisions in a stack were made and the context around them.

So, that emergency response, you're gonna get—like, in that example, the team will get it done, but then best practice would be to write an ADR, add to your ADR, add to the documentation, “Here’s what we did, here’s why we did it, and here’s why it was impactful.” And so, I've found, and the team, we found that it—while it was great to just document, over documentation is a crux as well.

Nick Koutrelakos: But when it really helped is when you're on boarding people and you have to just go through this tree of why we did this and all these decisions, and it’s very difficult to convey in a conversation. You often repeat yourself, all kinds of things like that.

Nick Koutrelakos: Having this documentation tree of not just, like, “Here’s what’s in place,” but also the what and the whys and the hows behind it. And having, and being able to direct new hires to, like, “Oh, yeah, this is a little bit unconventional. Read this, this is—you know, read this and then we can talk about it, and here’s why we did it, this is the emergency that was solved.” So, I think that having architecture decision records, those ADRs in place as a policy to actually maintain as your company grows is an important feature.

Brian Dawson: Awesome. And there’s a lot I can comment on that, but I think you said it all, and you said it well, right? Architectural decision records, documentation, standalone and as a response to emergency responses.

So, Nick, it was great to meet you, great having a conversation with you. Again, thanks for taking this time with us on DevOps Radio. Please take care of yourself, sheltering there at home. I do assume there’s probably another silver lining and that’s that you'll be getting a lot of cool learning and coding done while sheltering.

Nick Koutrelakos: Oh, yeah. More time—more time for learning.

Brian Dawson: More time to learn. Alright. Thank you much, we're glad to have you with us, and we look forward to talking with you again in the future. Bye, thanks. 

Nick Koutrelakos: Bye, thanks. Much appreciated, Brian. Take care.

Brian Dawson

Brian is a DevOps evangelist and practitioner with a focus on agile, continuous integration (CI), continuous delivery (CD) and DevOps practices. He has over 25 years as a software professional in multiple domains including quality assurance, engineering and management, with a focus on optimization of software development. Brian has led an agile transformation consulting practice and helped many organizations implement CI, CD and DevOps.

Follow Brian Dawson on Twitter.