Redox provides an API to healthcare companies for the thousands of legacy systems on which healthcare records are stored and must be processed. The ongoing boom in telemedicine and at-home testing has spiked a new demand for Redox’s services. Their engineers must be dually proficient in both APIs and event queue management, both to ensure accuracy and meet HIPAA regulations on patient privacy.
Announcer: Welcome to The Software Agents. Meet the people who bring software to every aspect of life, society, and business to handle the challenges of a transforming world.
Christina Noren: Welcome to The Software Agents, a new podcast on how software is helping the world survive and evolve right now, as told by the people who are making it happen. I'm Christina Noren, and my co-host is Paul Boutin.
Paul Boutin: Hello, thanks for tuning in.
Christina Noren: The Software Agents is sponsored by CloudBees the enterprise software delivery company. CloudBees provides the industry a leading DevOps technology platform that enables developers to focus on what they do best—build stuff that matters.
So, today, we have a great guest, James Lloyd, who is the Co-founder and CTO of Redox Engine, and we flagged this area. So, Redox Engine is involved in medical records APIs and he'll tell you more about that. We flagged this because a lot of the people inside of CloudBees, when asked what areas and topics they wanted us to cover on this podcast, we've had some people in our organization who have medical backgrounds, and they really flagged that medical records exchange is a huge issue with COVID and with trying to have distanced health care and the efficiency of care and the outcomes. So, we wanted to find someone doing something interesting in software and a little research brought us to James; he seems to be one of the more dynamic founders in this space.
So, James, tell us a little bit about yourself and Redox Engine.
James Lloyd: Yeah, yeah. Thanks so much for having me, Christina and Paul. Yeah, so, just to get started, Redox is a health care API and data platform to—and our customers really are folks who are delivering software within the health care space, typically trying to get their product in the hand of patients and providers and typically selling that directly to health systems. So, fairly enterprise side of things in health care, but some of our customers that you may be familiar with would be someone like Glooko who helps with diabetes management, folks like Healthify and NowPow, who are in the social determinants of health space. We work with a lot of folks who are helping with medical transportation, so helping patients get to the doctor’s office when their appointment is.
And, as you mentioned, as of late, a big portion of our business and our customers’ business is coming from things like COVID testing, telehealth, remote patient monitoring—those areas where more and more care is happening with the patients still in their home rather than coming into a physical health care location.
Christina Noren: So, tell us a little bit about you before we get more into Redox specifically. So, James, it seems you're more of a software person from your background, so how did you get into this particular area of software?
James Lloyd: I actually started my career and my college degrees and everything like that are in math and physics, and I was doing a lot of computational physics as part of my training. And ultimately, whenever I left or graduated college, it was 2008. Wasn’t an awesome time to be on the job search, but I did end up starting to work at a place called Epic, which is one of the largest electronic medical record companies out there. So, they make the software that doctors and nurses and the front desk staff and the billing staff use on a daily basis across a lot of the U.S.
And so, I got, from about 2008 to 2013 was when I worked there and saw a lot of adoption of electronic medical records and really got started with my career there and learned all about health care there on the job.
And around 2013, me and a few friends didn't quite know what we wanted to do, but we knew we wanted to be in a smaller company, be in a place where we can start our own company and really kinda control our own destiny and have a bit more of a personal impact.
And so, we started a coworking space in Madison, Wisconsin, where we lived at the time, and that coworking space went really well. We grew it pretty quickly to be Wisconsin’s largest coworking space. And throughout that process, we then saw that a lot of folks were coming into the coworking space with deep technology or health care backgrounds, but didn't really have the structure to really have an impact with that background and with that skill set. So, we built an incubator inside of the coworking space and specifically for health care technology.
And through that process, learned—we started about 10 companies and we learned a lot about what it takes to build and deploy and scale health care technology. And we learned about all the ways that that’s hard. And so, combining that with sort of a growing sense that there was going to be a proliferation of specialty products in health care based on some of the digitization that had come along with what we had seen at Epic, we really felt like this platform to help developers and technologists get their products into market and get them in the hands of patients and providers really made the most sense. That there was sort of a gold rush going on in digital health and our motto was to be the ones selling pickaxes and helping people kinda get to market and lower the barrier to entry.
So, that’s what we've been doing, and so, my role in all of this has been to kinda, I started in that incubator helping build and helping support the technology teams and those companies we were starting. And then, as we were getting started, really kinda led the Product Engineering teams through the initial build of the product and now, about six years in, I lead the Product Engineering and Security teams and we have about 150 people at Redox and—yeah, yeah, we're goin’ strong and yeah, I don't know, is there anything else you want me to comment on there?
Christina Noren: That’s great. So, tell us a little bit more about what the Redox platform actually does and what are the use cases for how these other systems integrate with it and, you know, and why do people who are building these other health care systems, these specialty systems need to integrate with something like Redox?
James Lloyd: The one really important part to know about the health care landscape as sort of context for this is, each hospital or each health system has, by and large, has its own data center and its own on prem medical records system that’s been highly configured over the past 10, 20, 30 years, however long they've been using that technology.
So, for folks who are from outside of health care, what I often compare it to is something like imagine you've been using Salesforce in your company for the past 30 years and you've been tweaking it and adding fields to it and adding drop downs to it and everything like that. And now imagine that there’s about 5,000 of those across the entire country, and that’s where all of our medical records live. And the other part that’s really important here is that, for the most part, medical data is exchanged through health care specific protocols and standards. And those are really hard to understand and there’s just a large learning curve for folks coming from outside of health care—not super modern and not super familiar compared to things you may have experienced across the rest of the ________.
Christina Noren: So, I remember in 2003, 2004, I had to get familiar with HL7 as an XML standard for health care exchange, you know, it’s sort of tangled with the ________ space. Is it still HL7, or what is it now?
James Lloyd: HL7 is both the name of the standard and the name of the standards body. So, there is a few versions in play that come from HL7 so, the most predominant one is HL7 version 2, and that’s what’s been around for probably 25 or 30 years. There’s an emerging standard called FHIR, F-H-I-R, which is a more object-oriented REST based approach that, you know, seeing some adoption in and actually has some upcoming support from regulation to drive some of the adoption up.
Christina Noren: The wonderful thing about standards is you can have lots of them. [Laughter] You know, so, there are all those. I assume that these customized 5,000 different systems in the country have developers who’ve been working locally on whatever transformations to emit in whatever standards. Why do you need Redox, then?
James Lloyd: Yeah, absolutely. So, one of the big value adds that we provide is the standardization across all of those different health systems. So, we basically author an API schema that is consistent in your interaction with all those different health systems. So, a software developer would need to connect to our API via our API documentation and that API schema. And then, on the back end, we normalize to that against all of the different configurations and local versions that exist in the health system. The other thing we do is provide a modernization of that, so you get fairly familiar JSON based structure that you can interact with, either through Query or through Webhook style event notifications. So, it’s pretty comfortable for any web developer to interact with.
And so, for our customers, what this means is, they can have members of their engineering team go from working on a health care integration to then working on either a backend project or a frontend project. Whereas, without us, when you're digging into these custom direct integrations to the health system, typically what that means is, folks are building teams around just those integration expertise. So, you know, it’s a lot more scalable and your team is a lot more dynamic when you use a platform like us.
Christina Noren: So, there are all these 5,000 systems and then you've got people doing whatever COVID test labs and whatever other systems, the insulin systems, all these systems. Presumably, there has to be a partnering aspect where each of the holders of medical records agree to integrate with Redox and make their records translated and available to developers through Redox. Is that the case?
James Lloyd: Yeah. Yeah, that’s right. So, the authentication and kind of authorization process that we leverage is actually, it comes through HIPAA, which is the, basically, the patient privacy act and really what that defines is something called a business associate’s agreement. So, the health system would sign a business associate’s agreement with any technology vendor that they're using to take care of the patients that they're responsible for. And that would include Redox as well as any of our customers.
And that is really the, it’s a very kind of B2B type agreement, it’s not like an individual patient providing consent, it’s more of the health system saying that this technology that we're using is critical to care. And the way that you as a patient actually consent to that is, whenever you first go to the doctor and they hand you that giant stack of forms, one of those forms say, “I'm gonna allow the health system to use tools that they deem critical to my care.”
Christina Noren: I ________ having to read every page of HIPAA, it was many years ago, so I remember this now. But does that mean that Redox has to contract with each, you know, contract according to HIPAA with each of those health systems to have access to the data?
James Lloyd: We work directly with the health systems and form relationships with their teams. The way these business associates agreements work is that you can kind of chain them together. So, let’s say—so, basically, when we're acting as a service provider to one of our digital health company customers, we have a BAA with them and then they may have a BAA with the health system. And then, in turn, Redox has a BAA with AWS or something like that. So, you can kinda chain them together in terms of who’s a service provider to whom.
So, there’s kinda two options for us. We can either go directly to the health system and form a direct BAA, which we do often, but in some cases, we're more of a subcontractor to the software vendor that’s providing the tools.
Christina Noren: Yeah, so, in this, we kinda try to understand the nitty gritty because we're trying to inspire people who are software people to realize that they can bring software and logic skills to different areas. So, it’s good to get the nitty gritty.
Has there been any impact to your business or to your software feature set or demands on your platform that have been noticeable in the last six, seven months of COVID?
James Lloyd: Absolutely, yeah, 100 percent. [Laughter] So, I would say the first thing that we saw in March—and I should clarify for your listeners, we're pretty much entirely U.S. based. So, the numbers and experience here may have varied if we were more international, but we were pretty much U.S. based. And yeah, so, in March or so, we really saw a strong slowdown, I guess, from health systems who were saying, “Really, the only thing I can focus on right now is addressing this crisis and this emergency.”
And so, we have customers that are providing tools to help with billing or help with reducing wait time in a clinic and some things like that, that just became a much lower priority for some of these health systems, given the kind of emergent situation that everything else was in. And so, the first thing we saw was just, health systems were kinda de-staffing—a lot of the IT projects just to focus on the critical nature.
And then, as maybe getting into kinda April and May and a little bit in June, we really saw that health systems were starting to then start to lean on technology to address some of those challenges. So, what that looked like for us is a strong kind of clustering of demand amongst that population of customers we work with that have, that are in the telehealth space, in the remote patient monitoring. Which, what that means is basically, you can think of sort of like IoT type tools that you may take home if you have a chronic illness and things like diagnostic in-home diagnostics as well all became, you know, we had companies that were growing 300 percent in a month type growth rates that happened to be in those types of verticals. So, delightful saw a strong demand there.
And now, I think the kinda meta topic that’s a little bit on everybody’s mind is, because—so, health systems by and large make their revenue from something called elective procedures. And the term elective is a little bit of a misnomer—it just means that it’s not kinda life or death in that moment. But, you know, you kinda opt into it. So, this could be anything from an MRI to actually having a tumor removed or something like that. So, it can be fairly critical, but typically, these end up being the high dollar type procedures, and many of those have been postponed. And so, the health systems are in a bit of a financial crunch right now. And that’s kind of having a bit of a ripple effect through the industry.
And I think furthermore, I think a lot of folks are, you know, from a kind of patient and consumer standpoint, a little reluctant to go to a care setting. Just, it seems like that’s where sick people might be, and so maybe you try to figure your way through it on your own a little bit more than you have in the past. And so, the thing that we're monitoring and looking at now is both the financial health of the health systems as well as this really being a catalyst for a bit more consumerization in the health care space. So, where I may have gone to the doctor to get a lab test, I may now search to see if there’s a mail order version and I'll just pay out of pocket because I don’t want to maybe take on that risk.
Christina Noren: Let’s get more into the software shop you run. So, with the sort of slowdown in demand in March and the concentration of demand on particular categories of customers in April and May and now looking at this consumerization, has that driven any change in your, in what features you're choosing to invest in and/or how you're running your software development or running the service?
James Lloyd: Yeah, yeah. So, I think there’s a few kind of industry specific or vertical specific components that we've added in. So, before—I'll say before 2020, the typical interaction across our platform was a software vendor who was connecting to a large health system. And then, in reaction to a lot of what was going on and in supporting our customer base, we've grown into supporting a lot more connectivity to state and local registries. So, there’s all sorts of kinda CDC and government run registries to help with tracking outbreaks and tracking results.
And so, we have a—we basically built a standard way to connect to all those state and local systems. And maybe this is not a big surprise, but government kinda registry type software is not the most modern thing to integrate with. So, we're providing an API in front of what ends up on the backend being some fairly clunky SFTP type stuff and fairly bespoke to each state as well. So, really allowing a lot of these diagnostic companies and COVID testing companies to be able to report to the states, all 50 states in under a month where that might have been a year-long project for their team, otherwise. So, that’s definitely shifted for us.
As a company, we've been distributed, so a remote first kinda company since we started. So, in terms of our operations, not a ton has changed other than now many of our folks have kids at home and other kinds of stressors on their lives just like everybody else does that we're dealing with. But in terms of kind of major shifts to our operations, not a ton has really changed for us.
Christina Noren: Yeah. We're hearing that from a lot of folks. And, you know, we're the same. At CloudBees, we have people in 17 plus countries and 16 or 17 U.S. states and we've been remote first, although we had a few people that were attached to offices. But we are finding that distracted distributed asynchronous remote work is different than your spouse and your kids are gone all day and you're just, you know, and your home is your office.
So, we are finding that people are identifying some areas where they still are more synchronous than they need to be or are missing automation because they're relying on being virtually present with each other all day. Are you finding any of that?
James Lloyd: The biggest challenge for us maybe in that space is just, you know, we've been remote, we've had a lot of that kind of automation and planning around how we work. And I think right now, just being compatible with everybody’s schedules and everything like that is just, we've just really doubled down on the kind of asynchronous decision making and driving documentation as the way of our major form of communication as we've got folks who are splitting time with their kids between their spouse and working kind of wild hours and things like that just to make it all fit together for their family, and being supportive of that, I think, has been the biggest shift for us.
Christina Noren: So, let’s get a little bit into the practice of software development for you. So, you own Product and Technology and Security and Development, so how do you decide what to build, how do you go through the process from definition of a feature through to delivery? What kind of automation are you using? Are there manual steps, are you doing continuous delivery? So, take me into the software factory a little bit.
James Lloyd: So, I'll start with sort of how we work from a product management standpoint and then kinda flow through that into the software development side. But in terms of how we operate from a planning standpoint, we're heavy consumers of sort of the agile model and try to get as much kinda continuous iteration and feedback as possible. We do try to drive kinda quarterly objectives through an OKR structure.
And maybe just to give you a sense of kinda what our team looks like, we've got—we've basically got five application levels, so end user facing kind of engineering squads. And then we have two infrastructure squads that comprise our R&D team. And each of those squads have their own kind of objectives and key results on a quarterly basis. And so, that kinda drives the kind of prioritization and the kinds of things we're trying to take on, but we're really trying to get as much into the ship something of value every week, every two weeks at least. And so, we do use continuous integration and continuous deployment processes. So, folks are pushing code to production on a daily basis.
And yeah, so, then in terms of the automation that takes place, we're using CI/CD tools, so we use Jenkins to do the code deployment and then running everything through Docker and Kubernetes. And then I think the other kind of important thing that we do is, you know, we have kind of a relative to maybe some other more user facing products, our real user experience is actually the API itself. And so, our code base is slanted probably a little bit heavier towards backend than maybe your average software company would be. And we were doing a lot of API requests per day.
And so, one of the things we roll out is what we call a blue green deployment or, as some people call it a canary deployment as well, but basically, this is where you may have kind of a horizontally scaled set of application servers and we roll out kind of progressively through those, and if we detect failures, it kind of automatically revers so that we, you know, the old version is still available in the majority of those application servers and we don’t have a disruption to our service.
Christina Noren: And how are you observing production? So, how are you detecting if there are problems with those rollouts?
James Lloyd: Yeah, so, we have all sorts of measurement tools. And so, we have kind of application level errors that we throw and then also just sort of the, we drive a lot of information through Grafana as well, and then we notify our folks through a tool called Pager Duty, just to have an on call rotation through that.
Is there a specific set of observations or anything you're interested in?
Christina Noren: We're trying to go from the why and the what all the way through to the how in these conversations, so just trying to get into different software shops and [Cross talk] tools.
James Lloyd: For sure, for sure.
Christina Noren: And obviously, being a Jenkins company, we like hearing that, so. [Laughter]
James Lloyd: [Laughter]
Christina Noren: Anyway, so, I’d love to understand a bit about the footprint of your service. It’s mostly an API, you know, as effective frontend and you're saying you're running in Docker and Kubernetes. You mentioned AWS. Are you running all AWS or are you running hybrid cloud?
James Lloyd: No, we're all in AWS right now, and in terms of kinda what our overall architecture looks like, we do have, like I mentioned, the vast majority of our kinda tech stack is based around API and message processing.
So, one of the maybe more complicated things that we do is that a lot of what we get from the health system or a lot of the way the interaction with the health system works is over a site to site VPN tunnel, which is what’s required for this HL7 version 2 communication. And it’s also all event based, so you would get a specific message for the patient’s been admitted, the patient got transferred from one room to another, the patient got discharged—things like that. That’s like an individual event that we get.
And so, in order to provide the normalization component that we do, while preserving the kind of FIFO nature of these events, we have to come up with some pretty creative technology. And so, we actually created our own queuing mechanism to be able to handle—to be able to basically run the normalization process in parallel for these messages, but then stitch them back together in order and send them out in sequence back to our customer on the other end.
Christina Noren: Is this in lieu of using something like a Spark or a Kafka?
James Lloyd: So, we do use Kafka between our services, but this is because we have to—we basically have these constraints of doing really fast processing on this normalization layer that needs to be FIFO and needs to be guaranteed to be delivered only once.
And so, as we started messing around with different queuing mechanisms, we never really got to quite the point at which we were satisfied with what we were able to get out of the box with any of them. So, for this specific component of kinda running these normalization configurations—yeah, we had to create our own queueing mechanism for that.
Christina Noren: So, James, where do you see health tech going through the end of this crisis and in the years following?
James Lloyd: I alluded to it earlier. I think there is a lot of consumerization that is being accelerated through this time right now. We've even had regulation come out that, in support of telehealth, that didn't exist before. So, there’s definitely both a consumer demand as well as a just industry shift towards more distributed services. And I think one of the things that will be interesting to observe is just how much this becomes a more online, more digital type experience.
The analogy I make sometimes to compare to something outside of health care is 10, 20 years ago, the mall was the place to go buy everything and today, something like Amazon is the place to go buy everything. And in health care, we're very much used to operating in this situation where you go to the clinic, you go to the doctor’s office, you go to the hospital to get everything. And I think we're gonna shift towards a lot more being delivered by mail or being delivered virtually.
So, I think for us and for the companies that are being created right now, having your affinity towards the patient versus towards the health system I think is a big component of the shift that’s gonna happen, where you may have had, you know, thought about your accounts or how you would structure your user permission or anything like that around a specific organization. I think now, it’s gonna be focused around the patient and the doctors that may need to know about that patient’s record may be in a variety of different organizations, and the affinity is gonna be much closer to the patient.
Christina Noren: Do you see me owning my health record and deciding where to house it and all of my providers and systems I interact with have to update where I've chosen to have my master record? Do you see it going that far?
James Lloyd: I hope so. [Laughter] That’s what—I think that’s the dream for us and that’s what I would really hope that we can empower and that the industry lands on. I think the reason why that hasn’t been the case so far is just because of—you know, there’s certainly some consumer economics and some political things, but at the end of the day, the technology component isn’t there to support it even if it were something that made sense from all those other sides.
Christina Noren: It seems like you’d be in a position, you know, if the political will were there, it seems like you’d be in the position to be a provider of personal health care record master hosting or whatever you end up calling it.
James Lloyd: Yeah, yeah, and that’s definitely something that we talk a lot about and aspire to provide, for sure. I think there’s some interesting challenges both on the identity component as well as actually the medical data content side. So, kinda mentioned earlier that the databases are all sort of fractured between all these different health systems and there might be three or four different Christinas across different health systems and there’s actually no master kind of mapping between all those to say that you're the same person anywhere. So, that’s the first challenge and that’s the one we're actively working on right now.
Christina Noren: It’s almost like in sort of enterprise software relationships, if you ever follow, Doc Searls went on for a while about vendor relationship management and taking ownership of your vendors instead of the vendors owning customers with CRM. So, it seems like there’s some similar dynamics there, but I digress.
Paul, do you have any thoughts of—you know, you've been listening and what’s your big takeaway from today?
Paul Boutin: Oh, first off, this is great. I have worked with health care related companies and I did six months at USC’s Keck School of Medicine, so a former Google engineer turned M.D. at L.A. County Hospital said a lot of things are an afternoon to code and two years to get it approved and rolled out.
James Lloyd: [Laughter]
Paul Boutin: When I hear API and HIPAA in the same sentence from somebody, that’s where the solution’s gonna come from. To be a software agent, you can’t just be a software person, and you can’t just be a health care person, you have to be both in the same head. And that’s what I hear in you. And it comes out as, we have to make our own queuing system, but at the same time, what many of these companies talk about is becoming more patient focused. Everybody’s frustration with health care is often that some set of companies that you don’t even know who it’s going to be when you get started on a health care journey have all these different rules and ________. And I found myself, there were two of me. My middle name was spelled differently in records, so there were two of me.
So, this is really great, and it seems to me like it’s really just a matter of having somebody who knows how to do an API and message processing the right way, but also understands what patient focused health care, especially now that we can’t just go to offices all the time, is going to mean. You've explained it very well.
James Lloyd: Oh, thank you.
Christina Noren: Well, I can’t add any more to that summary, and I'm just going to thank you, James, very much for taking a leap and talking to a few strangers about what you're doing, and I think we're really excited to keep watching Redox.
James Lloyd: Absolutely, yeah. Thank you for having me, and thank you for doing this podcast. It’s a great opportunity.