The Software Agents is a new podcast series sponsored by CloudBees. Each week we bring you leaders from many fields applying software to reimagine them for the new world under construction.
Running race events worldwide were canceled overnight in March. RunSignup founder Bob Bickel explains how the company's DevOps DNA enabled them to survive the sudden shutdown of their core business and create new kinds of events that have kept the running community together and still running.
Announcer: Welcome to the Software Agents, a podcast series that brings you the people using software to help change the world in this time of transformation. With the help of some of the brightest technology minds, we'll explore how almost every possible area of life, society and business is being reimagined through the power of software.
Christina Noren: Welcome to the Software Agents, a new podcast on how software is helping the world survive and evolve right now, as told by the people making it happen.
I'm Christina Noren, and my cohost is Paul Boutin.
Paul Boutin: Hello.
Christina Noren: Software Agents is sponsored by CloudBees, the enterprise software delivery company. CloudBees provides the industry's leading DevOps technology platform that enables developers to focus on what they do best: build stuff that matters.
Today, we have Bob Bickel, who in addition to being a CloudBees board member and a great mentor for all of us here – is a long-time software leader and innovator who's built many of the companies that are foundational to how we build software these days. And these days, is the CEO of RunSignup. I'll let Bob tell you a little bit about himself and what he does these days, and what RunSignup is, and give us some context.
Bob Bickel: Thanks, Christina. Thanks for having me on.
RunSignup started in 2010. It was really this collision course between my passion for technology and my passion for running. They just came together and I saw a need in the marketplace for races to utilize technology better for people to sign up and share in the moment, and the glory of setting their own personal records, or finishing a half-marathon or a marathon.
Felt like there was better technology needed, and so we started building stuff and it's worked out. Last year, we had over 21,000 races across the U.S. use us to sign up over six million people and deliver over eight million results for those people. So, it's been a fun ten years.
Christina Noren: That's amazing. I've always enjoyed hearing you talk about it, and also talk about the software delivery behind it.
But I'm guessing that in 2020, you're not gonna be running 21,00 races. So, what's happening right now?
Bob Bickel: By mid-March, our business had fallen 80 percent. Basically, pretty much all real races came to a halt. There was a lot of uncertainty in our own business. Would we survive? How would we deal with that?
We actually cut expenses not by cutting people, because we have confidence that our business is gonna come back, but by all sharing in it together. We actually cut everybody's compensation by 50 percent.
Christina Noren: Wow.
Bob Bickel: Fortunately, the PPP program came along and we were able to reestablish 100 percent pay for our people.
Also what's happened is that the world changed and opened up new opportunities for us that we were able to take advantage of.
First, it was virtual races. Then, the second one is a whole new category called challenges. For example, running or walking across the state of Tennessee during the summer, basically recording your miles and seeing yourself move along the map during the course of the year.
Between all of that, our revenue is still down, but down only 20 percent. So, it's a very manageable situation, where we can carry forward and do good work.
Christina Noren: I want to dig into that a little bit more, but before we go there, I always like to give people who are listening a little more context on the person they're hearing from. I'm gonna ask you to talk a little bit about your background and your journey through software into here, and what enabled you to get to the point where you can start and run this.
Then, we're gonna dig in a little bit more into how RunSignup is evolving for these new use cases that are still valid in this time. So Bob, tell us about you.
Bob Bickel: One of the important things to know for the running side is college, I actually ran, and I was a semi-decent runner. So, it's kind of stayed with me and it was a foundational part of who I am, about learning how to work hard and be part of a team, and so forth.
In the early '90s, I was very fortunate. I was working at a small consulting company, running this little software group, and stumbled upon this little thing called the Internet and web.
I was one of the early adopters of the idea of putting applications on the web, basically tying databases to the web. Got the wonderful experience of helping Bluestone Software grow, take venture capital. We went public during the boom year of 1999. Stock went from 15 to 130, back down to 15, and we got acquired by Hewlett-Packard.
I wound up running the middleware division of HP for a year or so, and then Hewlett-Packard changed strategy and went into the PC business rather than software. So, unfortunately I had to close that division down.
I left and started working with small startups. Because I had had this specialty of Java infrastructure type of stuff, I was able to hook up with one or two entrepreneurs at a time –
and help them through the process of growing their company and trying to find product market fit, and so forth.
I got really lucky. I hooked up with JBOS, I hooked up with Spring, I hooked up with Hyperic, hooked up with Princeton Softtab, hooked up with Jaspersoft. All those companies went on to some degree of success. I was just kind of along for the ride, just helping as it were.
In 2010, I did the same thing with CloudBees. My friend Sacha Labourey started the company in 2010. He reached out to me for ideas on the company and so forth. So, I became a part-time consultant for them.
Christina Noren: Yeah, you guys had worked together at JBOS, right?
Bob Bickel: Yeah, we got to be friendly at JBOS. We see the world very similarly, even though he's Swiss and I'm American. I'd say we love each other.
Christina Noren: I can always see that.
Paul Boutin: I think we should point out that Bob still has a role with CloudBees.
Christina Noren: Yeah, I did at the beginning. I said you're still a board member.
Bob Bickel: Yeah, I'm still on the board. I'm an investor and an advisor to the company. I will take credit for also having known about what was called Hudson then, and Jenkins, and this idea of continuous integration back in 2010.
I was the person that did the introduction of including Hudson into the CloudBees platform.
Christina Noren: Wow, that's _____.
Paul Boutin: I have one more newbie question. I think it's important. Bob, with all that background, this is another case for our show where you looked at running and said, there are things missing here. What's the two to three elevator floor version of what RunSignup brought in that you saw missing?
Bob Bickel: What really bothered me is that big races could have decent technology. Like the New York Marathon, you could track runners and have results, and things like that. Little races couldn't, and it didn't make any sense to me. I come from this open source world and I'm like, people should develop open technology and make it available to everybody.
So, that's basically the premise of what we did. We decided to build just good technology, make it very inexpensive, very easy to adopt, single platform, SaaS platform. Now, a 5K can have all the features the New York City Marathon can have.
Paul Boutin: Terrific!
Christina Noren: Yeah, it's funny, Bob. That's the same thing that drives me nuts about the museum app world, and we had a guest around that recently. It just makes me insane that the biggest museums develop their own individual apps and there's no reason to have a museum branded app, and all these other institutions don't have what they need.
So, I share your frustration in a different domain.
Bob Bickel: Yeah.
Christina Noren: Let's get back to RunSignup today. You had to make obviously a lot of software feature changes, or at least I imagine, to adapt to these new contexts of these challenges and so forth. Tell me a little bit about what those changes were and how did you rally the team around them, and how did you make it happen?
Bob Bickel: Well, to give a bit of background. I was actually very influenced by this idea of continuous integration back in 2010. So, when we started building RunSignup, we built it as a single platform, and we made it such that it was very easy to do software releases and not impact the running of the service at all. Since 2015, knock on wood, our site has not been down and we average about 1,500 releases of the software per year.
Nobody ever notices. We do upgrades to the database, we do new releases of little features and big features. So, it's made us a very adaptable type of company. What's differentiated us is our speed of being able to develop software and get it out the door to our customers.
It's this whole idea of continuous delivery. You make a bunch of small little changes, you get feedback from the market, and you continue to make changes into the future.
We were actually built for this type of pandemic, where a lot of our technology needed to shift and change. There were three major things that needed to shift and change.
One was on the financing side. We move about $270 million of money per year from runners that want to sign up for races to race organizations and nonprofits. So, we needed to rewire some of that to accommodate for refunds and chargebacks, and things like that. Just to make sure that our fundamental purpose in life, which is moving money, we held up that end of our bargain.
The second area that we did is virtual races, and the third is this whole new platform we developed from scratch called challenges.
Christina Noren: Before you move on, just for a little bit of context, how many people in the organization? If you don't mind sharing, how many people are directly involved in software delivery?
Bob Bickel: We have 50 total people in the company. We're a small, very efficient company. We're actually employee owned, so everybody in the company has a share of the company and there are no outside investors. The development team is 15 people, and then there's a couple of product managers in addition to that.
Christina Noren: What I heard is you had to deal with your financial infrastructure, you had to adapt to virtual races, and then the challenges. Let's take those a little bit one by one.
On the financial side, the chargebacks will kill you. I can only imagine. Are you taking the risk on the chargebacks, or are the race organizers taking it? How did people deal with the financial impact? I know personally, I'm still waiting for refunds on events that I paid for and couldn't go to in March.
Bob Bickel: Yeah, it's a combination of all that. We're responsible and the race organizations, the nonprofits, are responsible.
We're what's called a payment facilitator. You can think of us like a branch bank. Each time a new race or nonprofit organization comes to us, we have to underwrite them and create what's called a sub-merchant account in the credit card network.
We manage the movement of all that money. Previous to now, we kept zero reserves. People signed up for a race and every day, or ever week, or every month, depending upon what the customer chose – we would move that money over into their own bank account. But the way it works is that if a chargeback happens, that race or nonprofit is responsible for paying that. We have no way of pulling that money back from the customer, so we would wind up holding that money.
On March 9th, we had paid $75 million to races that had not yet happened, and had next to no reserves.
We went down two parallel paths. One path was to provide our customers a whole bunch of tools to be able to communicate with their customers.
The endurance community is kind of an interesting community, as are the nonprofit customers that we serve in that the people who sign up for the races or the people who make donations, typically there's some affiliation there. There's a feeling that's deeper than just buying a shirt on the Web, or something. So, what happened is that the race organizations cared about their customers – and they started putting back money into our system. Then, the race organizations offered a slew of different options to their customers from deferrals to virtual races to refunds to partial refunds. There is really honest communication. It's kind of a weird world, but races would say hey, we've already spent half the money, so we can only give you a 50 percent refund.
The customers were okay with that. It's a really great environment.
Out of that $75 million, I think somewhere around $8 million has been refunded. Less than $100,000.00 has gone through chargebacks, and 98 percent of that has actually been paid for by the race and nonprofit organizations.
Christina Noren: So, your merchant account is safe?
Bob Bickel: Yeah. It wound up, I was losing sleep in March, Christina. I was really worried.
Christina Noren: I want to dig into [break in audio] for the software perspectives. Sorry for interrupting. From a software perspective, you've got this team of 15 developers and 2 PMs, and the rest of the people in the organization. Like all of us, you're reacting to this fast-moving situation, no pun intended.
How did you, as a team, come together and figure out what changes you needed to make? You obviously built these communication tools around these refunds, and chargebacks, and status. We all know that suddenly coming up with new features you hadn't been thinking about and getting them built is not just a matter of technically being able to update the site, but it's around getting humans to figure out what the specs should be.
How did you get the team together around that, and how long did it take to start getting these features in race organizers and customer hands?
Bob Bickel: Pretty quick. We're really lucky. We have a couple of especially talented software developers and our whole team is, I'd say, above average. Then, we have a very tight organization.
I read this article on Friday, March 6th. It was in The New York Times and it was by this mathematician from the UK. I finally understood what was gonna happen. I came in on Monday and I was like, we've got to freeze everything we're doing and start doing this new stuff.
So, we had these daily huddle sessions, where we'd understand what was happening from customers and in the market, and where our balances were, and all that. We had finance, and product management, and sales, and development all on the calls.
Then, we would prioritize what things do we need to get done today? So, we put out a bunch of releases that week of March 9th. It was incredible how much effort everybody put in. Our CTO, Stephen, probably worked 18, 20 hour days during that entire week, getting stuff out the door. It was really amazing what we were able to put out.
By Wednesday, we actually had a tool for our customers, a postponement/cancellation tool, that allowed them to basically put a pop-up on the front end of their website that gave customers choices.
So, it provided this immediate facility to allow races' customers to communicate with the race, and for the race to communicate to the customers very explicitly.
You could push a button as a runner and say yeah, I'm gonna sign up for the virtual race, or turn my race registration into a donation. So, there was this engagement that happened. We got that done between Monday and Wednesday.
Christina Noren: That's certainly a validation of the agile methodologies.
Bob Bickel: Yeah.
Christina Noren: The next one you mentioned was the virtual races. Was that a brand-new feature, or something that you were already doing? When did you start to have that kick in and what did that look like, from the software?
Paul Boutin: What is a virtual race, for those of us who don't know?
Bob Bickel: A virtual race is a race that happens virtually, Paul. As an example, the race that really started RunSignup was called the Scowcoffee Race in Moorestown, New Jersey, my hometown. It's a race that benefits the Rotary. It's about 1,000 people every year.
We run down the main street of our hometown and do a loop and stuff. This year, it turned virtual 'cause we couldn't hold a real race. It just wasn't safe.
So, what a virtual race is is people still signed up for the race. They were able to run wherever they were and basically log their time, if they wanted to.
Upload their own photos, if they wanted to. All those results showed publicly. I walked the course 'cause I'm an old broken-down runner now, but I could still say that I completed the course. I could still post some photos, some selfies and things like that, trying to be funny on the race website.
Everybody got a shirt, and it was a sense of community. The shirt had a special design of people running with masks on, and things like that.
It's just a way to have a race carry on, but done in a virtual way.
There's also an app that we have called RaceJoy that allows people to actually run the actual physical course, or to run anywhere. So, somebody could run in California and get alerts on how fast they were going. We had some special messages in there that would play to people, like when they were going by the high school and things like that.
Christina Noren: Was that functionality already built before the epiphany week of March 9th?
Bob Bickel: About 25 percent of that functionality was built. We have had people do virtual races on our platform before, but it was never a very big part of our business, so we hadn't really focused on a lot of features. It had never really happened at scale.
Now, we have thousands of virtual races that are happening on our platform with hundreds of thousands of people signing up for those virtual races. Today, about 50 percent of our transaction volume is from virtual races.
Christina Noren: It's a funny community pool aspect, if people are still paying for that. I have a Peloton here and I'm used to working out virtually with other people.
I know people in the SoulCycle cult and they're just suddenly moving to at-home bikes, trying to ride with each other and maintain that community. There's an interesting social dynamic there.
Bob Bickel: Yeah, it is. It's a way to keep healthy. It's a way to still continue to support... There's a lot of people who like the Rotary and are connected to it in some way, shape, or form. It's a way for people to share in this really weird time.
Christina Noren: Moving to the last point you made, which sounded like it was the most brand-new thing, the challenges. Paul, you'll probably remind me to ask what the hell a challenge is, right?
Bob Bickel: A challenge, this started in April. This guy Laz, who is the race director for this really weird race called the Barkley's Marathon in Tennessee – decided that he would put on a challenge to run across the state of Tennessee. Laz does all these ultra trail race races on our platform.
Christina Noren: _____ _____ people that run like 100 miles at a time.
Bob Bickel: Exactly. It's crazy. Or see how far you can run until you collapse and die. Laz decided to put this up. He's developed quite a cult following. But he thought he'd get like 200 or 300 people to sign up for it. He had over 20,000 people sign up for his Great Virtual Race Across Tennessee, which basically people were going to run across the state of Tennessee virtually, and log their miles every day. So every day, they'd move a little bit further and things like that.
He thought he'd have 200 people sign up and it just blew up. We were like, holy cow. This could be something that's really interesting. So, we started developing all this software to help people keep track of their activities.
What you can do is you can come into the system, you can upload your Strava or whatever type of fitness app data file, or you can just enter in how many miles, or how much elevation, or what activity you did on a daily or however many times per day.
We keep a log of all that. We add it up, and then we show a set of results ranked in order of number of activities, or distance, or elevation, or whatever.
We also have a map feature. A race can upload the course from one end of Tennessee to the other end of Tennessee.
You can have logged, say, 53 miles and it will show you where you are on that race across Tennessee.
Christina Noren: You started building this in April because of this Lazlo guy starting to try to do things like that with your existing platform?
Bob Bickel: Exactly. There was all this functionality and we have had just a blast building it out.
Now, we're at a point where you can upload photos every day, you can get badges along different milestones. We have this feature that we're starting to work on right nowhere you get certificates. So, a certificate could be a graphic image.
One of the ideas that we're working on right now is that you would get a postcard. You're going across Tennessee. When you hit Memphis, you get a postcard from Memphis.
It has real-time data coming from the database, like how many miles you've run and your name, and things like that. But it also has texts in there like information about the city of Memphis, and links to videos. People are doing just amazing things. There's a Florida Keys race that's happening, where they're creating a video for each milestone that people reach.
Christina Noren: Wow. You've got a very productive team. Let's go back into the software delivery side, since we've gotten into the "what" of what you've been building. Clearly, the practices you built in from the start are what have enabled you to respond so quickly.
What does your infrastructure look like? What does your software delivery pipeline look like? What does your daily practice look like?
Paul Boutin: We're on Amazon. We've got a very multi-region, multi-availability zone set up. We use a lot of Sqs cueing. There's a lot of fault-tolerant features built into the system, which is why we survive downtimes. When Twitter went down a couple of years ago, we actually stayed up.
Christina Noren: No farewell for you?
Bob Bickel: We've got a very robust infrastructure there. We have to be PCI compliant for credit card processing, so we have external auditors that come in every year and audit us. We've got a very robust infrastructure.
How we do software development. We use GitHub as our source control.
Actually, everybody in the company has an account on GitHub. Every sales representative, and account manager, and support person has a GitHub account. As they interact with customers during the course of the week, they will run into either issues or ideas and suggestions for how we can make the software better. They actually log that as issues in GitHub.
Every Monday morning, our CTO, Stephen, Ryan Snell, who's development manager, and myself get together and we go through those issues. There are typically between 15 and 30 issues that come in per week. We categorize those. We either put them into a milestone, so a major milestone like the volunteer system. Or, like CORAL management, or something like that. Or, we decide that they're what's called a Monday task.
Monday tasks are typically things that will take less than an hour, and they're relatively easy to get done. Adding a column to a report is a good example.
We then put those into what's called the Monday task bucket. Developers will pick tasks up out of the Monday task bucket and will typically turn a bunch of features around within a day or two. A customer will have asked for a new field on a report last Thursday and on Tuesday, they'll see that field.
Christina Noren: I imagine that there's a really good happy competitive feedback cycle that probably plays into the kind of people who would work at a running startup.
Bob Bickel: It's fabulous in a whole bunch of different ways. Imagine a customer asking you for something, and they get it the very next week. Customers just aren't used to that.
Christina Noren: The handling of requests for enhancement is one of those arts. I've found it's like you almost want to optimize for the two extremes. You want to delight people with some things that suddenly appear right away, when they would have expected it to take weeks, or months, or a year. Then, you also want to keep the communications on the long pull ones, and surprise somebody that something that they asked for three years ago suddenly shows up.
Bob Bickel: Yeah. Then, what we do on the milestones is we typically give a single developer a milestone project. For example, Phil right now is working on photos, or Alyssa is working on donation websites. What they'll do when they start that milestone is they'll typically be scheduled to work on that for between a month and three months.
They'll go through all the GitHub tasks that we've collected over the past two years on photos, and they'll look at that. I'll meet with them, or one of the product managers will meet with them, and they'll prioritize and figure out what's easy to do, what's hard to do, what has a lot of benefit. This idea that came in 18 months ago is no longer a good idea. This idea is a crummy idea, but what we really should do is this other thing.
Then, we come up with the project plan. Then, the developer will just work on it and incrementally knock down issues. They typically will put out a dozen to two dozen releases during that one to three months, so they don't build it all up and release it all at one point in time. So, we're very incremental in how we do our releases.
The other important part of it from a personnel point of view is that having a developer responsible for something makes them very energetic about it. The fact that they're reading these GitHub issues and the issue came in from Emily, they can reach out to Emily and say, what do you mean about this issue? Then, Emily can clarify that to them. The developer winds up being very connected to the customer in that way.
It's very efficient 'cause the developer is not winding up talking to customers and everything, but they're really feeling what's happening out in the marketplace. That level of personal responsibility makes them just put that much more energy into their development work.
Christina Noren: I find that as a PM, that sort of approach to getting engineering real ownership and incorporating customers feedback is fascinating.
Bob Bickel: The developers are so proud when they do releases, and the Emilys come back to them and say oh, this customer was so happy. Thank you so much. It just is such a positive feedback loop for everybody.
Christina Noren: It's funny. It's easier to do that when you instill it from the beginning, that you reengineer that to a team that hasn't had that.
Bob Bickel: Yeah.
Christina Noren: Continuing a little bit here. I wanted to get a little bit into the shift of software development practice. From what I've seen on Zooms with you over the years, you're used to working in an office with everybody working in an office. How has the team adapted to presumably working from home and keeping that cohesion?
Bob Bickel: We've always been semi-remote. My history is in open source and in open source companies, the development team is typically very distributed.
Most of our development team comes into Moorestown, New Jersey, but we do have a few remote people. Even those people who work out of Moorestown, New Jersey, only a few of them work here every day. A lot of people work from home or they've got their own place that they hide out. So, we're kind of used to working remotely.
We're a very technical company. Everybody's kind of a nerd and a little OCD, and all that. It's okay to be remote. So, it really hasn't been that bad of an impact for us.
Christina Noren: Are you pretty asynchronous, or do you have a habit of daily synchronous standups and stuff? Or, does any of that _____?
Bob Bickel: No, we're very asynchronous. Monday is development day, so we have that hour, hour and a half where Stephen, Ryan and I get together and go through all the issues.
Then, we have about a 45-minute what people refer to as a standup, where all the developers go around the room and ay what they did last week, and what they're gonna do this coming week.
Christina Noren: So, you keep the meetings pretty light?
Bob Bickel: Pretty light and pretty short. Part of the design process is that you want to minimize meetings because they're pretty inefficient.
The ideal situation is just have a single person that does everything. But if you can't have that, what you want is a clean set of APIs between everybody. You do your job, I'm gonna do my job. We need to send messages back and forth every once in a while, make it a very clean API and we don't have to have meetings.
Christina Noren: It was funny. It was memorable. When we were at an event together, you were talking about RunSignup.
After years of traditional venture-backed startups, you had a certain pleasure in being able to say that you had a company with no outside investors, which let you run things maybe a little bit more dictatorially.
Bob Bickel: Well, I like to think of it as a benevolent dictator.
Christina Noren: I can't imagine you being anything but benevolent.
Bob Bickel: There we go.
Christina Noren: I think we're coming close on time. Paul, what's your takeaways from this conversation?
Paul Boutin: As someone who literally is unable to run, I am surprised to learn that running is much more of a community activity than the person we see out there by themselves. Also, I suspect that these big races weren't trying to have exclusive software that no one else can have this feature. What they don't understand, and this is a big opportunity, is for people who get software, and get continuous development, and get open source – could go into other fields and say do you know what you could have? Do you know what everyone could have? I think that's great that anyone can have a race and not have to have this huge budget in order to have all the things you talk about. But also, I will say the amount of behind-the-scenes infrastructure, so that everything just works, is really amazing. It's best when no one really knows how much work you're doing. It just works.
Bob Bickel: Yeah, that's the thing. I call us the Goldilocks of endurance. If we had been a bigger organization, we wouldn't be able to focus and make purpose-built software for this marketplace. If we were smaller and didn't know how to run a development organization, we wouldn't be able to develop the complexity of software that we have.
Our competitors are either too big or too small, which is why we've grown so much and have so much market share.
We're actually launching a new business called GiveSignup. The nonprofit space has many of the same attributes. There's big companies like Blackbaud and Eventbrite that serve that community, but they don't serve it well. They don't serve it with purpose-built software that really tries to encompass the real needs of the customers.
They just give surface level-type credence to development. Technology is so core to any business. If you know how to build it well, and you know how to build it fast, and you know how to listen to customers, then you have this really competitive advantage in whatever market you're serving.
Christina Noren: It's funny. My takeaway from this conversation is a little bit of a tie to the last one of these we did, which is I think there's a little bit of a pattern developing amongst our guests, Paul, where really skilled developers who cut their teeth on building infrastructure software and laying the foundations for how we do software development today are finding some passion area in starting a bootstrap startup with minimal outside investors because they don't need that anymore, after their successes – and building those really hardcore, efficient software practices into their new small companies in their passion area. It's like the same story as John, so it's kind of funny, isn't it?
Bob, I think you'll enjoy hearing John V's recording from last week, as well. Kind of the story of bringing the skills from development tools and infrastructure over to let's apply it to something we care about.
I think we'll close there. Thank you very much, Bob, for the conversation. It's nice to have a lot more color on the things that _____ _____ _____ over the years.
Bob Bickel: Thanks. Always fun to talk with you, Christina.