The Software Agents - Episode 12: Quizlet - Natalie Rothfels, Senior Product Manager - How EdTech is Shaping the Workforce of the Future

The career landscape was already changing before the pandemic. Now, Quizlet is developing study guides for high-stakes certifications like EMT or CompTIA, while also supporting an explosion in home-learning content graded with the help of machine learning. Natalie Rothfels talks about Quizlet’s new prep packs, and recounts her journey from language nerd to software product lead.

Announcer: Welcome to the Software Agents. Meet the people who bring software to every aspect of life, society and business to handle the challenges of a transforming world.

Christina Noren: Welcome to the Software Agents, a new podcast on how software is helping the world survive and evolve right now as told by the people who are making it happen. I'm Christina Noren and my cohost is Paul Boutin.

Paul Boutin: Hello. Thanks for tuning in.

Christina Noren: Today we have Natalie Rothfels, who is a product leader at Quizlet, which is an interesting company that delivers a lot of online learning and fun education that is used by many, many children and teens across the world. I got to know them a few years ago actually through one of our previous guests, Karen Sun, who was one of our episodes a couple weeks ago, and this is kind of the Marvel Universe of software superheroes so one guest leads us to another, and Karen told us Natalie was a definite superhero software agent. So Natalie, tell us a little bit about what you do at Quizlet and what Quizlet is about.

Natalie Rothfels: Thanks for having me on the show. I'm looking forward to our chat. So Quizlet is a study platform that millions of students and teachers from all around the world come to every day, every month, to learn and master whatever it is that they may be studying, and that could be they're in a classroom, in a middle school or right now a Zoom classroom remotely, that could be they're in high school or college learning, in a more independent study session kind of way, or it could mean they're lifelong learners who want to catalogue their favorite birds of Northeastern Argentina, which is one of my favorite things to reference on our site. So Quizlet is a user-generated content study platform and people come to it to help them learn.

Christina Noren: It feels like there's a lot of software platforms like Quizlet that kind of grew up in a nice-to-have-way, and I can only imagine that it's suddenly a lifeline in this time. So what have you seen inside of Quizlet in terms of changes in patterns of use both in terms of content creation and participation in the last five months or so that people have been stuck at home and has that influenced what you're doing at all?

Natalie Rothfels: We're learning like the rest of us about what's going on in the world. I should say we're used by two in three high school students here in the US, but we're also used globally, internationally, in over 120 countries, and so what that means is that we're monitoring this global pandemic on our global platform. So we've seen shifts in both the traffic that's coming to our site, where they're coming from, but also how long they're staying around based on completely different patterns of what's happening in schools, right? So right now we're not in a time where every single school is following the same closure or reopening standards, we'll call them, or rules, and even those that may be following some may change them in a few weeks, and so we're really needing to be pretty on our toes about what's going on in our site and how is it different from what we're used to seeing at this back-to-school time.

As an example, we're in back-to-school right now in sort of North American countries, which is not every user on Quizlet or every student, and we're seeing kind of a different population of folks coming to our platform. They may not be as used to Quizlet, what we do, they may not have used it in all of their studies before, and so we're needing to be responsive to that in helping them get on the platform faster, understand what the product does and how to use it, and that's a real challenge of figuring out how to both stay focused on what our plans might've been or were and also be responsive to the ever-shifting dynamics of this current situation.

Christina Noren: Kids are coming back to school in North America and you're basically implying with which you said that they are starting to use Quizlet. Is that because teachers are pointing students at Quizlet? Is it because parents are leaning on Quizlet or is it kids discovering it on their own?

Natalie Rothfels: It's a mix of all of those things. I think we do have a lot of, I'll call it, kind of class-driven usage. So a teacher may bring all their students onto Quizlet for learning activities or even to assign things out as homework, and as classes are starting teachers are facing a lot of challenges of first of all how to get students online, second of all how to engage them in the types of activities that they may have been used to doing in a physical classroom where they're trying to differentiate for each student what activities they're doing based on different levels, and they're also reading the room of who's understanding what and how well are they understanding that and trying to evaluate engagement. We've also got teachers who are trying to figure out and districts trying to figure out what is the role of homework right now, what's the role of assessment right now?

And so I think we are seeing folks come on who are trying to be experimental, right, teachers trying to adjust to this new reality, but we're also seeing students needing to be more self-directed, which is I think a real challenge. I don't know how long it's been since I was in school, but it's been a handful of years, right? And so thinking about maintaining a sense of focus, attention, prioritization of how I should be spending my day, actually doing deep thinking and learning; all of those are just massive challenges for everyone right now, let alone I think for students and teachers who have never worked in this paradigm before either. So it's a mix.

Christina Noren: A lot more questions down that path, but before we go too much further tell us a little bit more about you personally and how you came to Quizlet, what your role is day to day and what your path to it was.

Natalie Rothfels: It's been a fun and winding path, but my whole background is in education. So I started tutoring when I was in high school and just finding younger students that I could help with schoolwork. When I went to college I was really interested in education. I thought maybe I would become a teacher and I started working with a program called the Neighborhood Schools program where I went to school on the Southside of Chicago and started teaching within high schools on the Southside, which was a really eye-opening and humbling experience for me to see the differences of urban education and how students were being treated, looked at, evaluated, all of that.

And I then moved into teaching after I graduated in a variety of contexts – a bilingual preschool, a university in Turkey – so my background is in education and I think part of my interest in that has always been what makes a good learning environment and how do we evaluate what that looks and feels like? I think that we all know the feeling of being engaged and excited about something that we're doing, but we also know the feeling of not being that way, and I think a lot of my school experience was a big mix between those different types of environments. So I was always really curious about what does it feel like to feel engaged, empowered in a classroom, and I think that's kind of led me down that path.

By total happenstance I happened to work across the hallway of an education company called Khan Academy after graduating from college, and I started meeting some of the folks who work there. Khan Academy is a nonprofit that also provides amazing learning materials primarily for K-12 students both here in the US and also internationally, and just through talking with those folks who were doing such mission-driven work I got really interested in it and I started using the product myself just to kind of relearn math and remember all the interesting bits of completing a square and things I had just totally forgotten. I got kind of obsessed with the product. So when I was living and working and teaching abroad in Turkey I decided to reach out to the Khan Academy team and say, "Hey, it would be awesome if you all could have this product available in other languages and other countries," and I ended up moving back to the States and working on that at Khan Academy. So that was my first foray into technology really.

I mean I was a classic Neopets Millennial user. It's not like I hadn't used any, but I'd never really been exposed to ed tech or tech as a business or as a product or something that I could be a part of building.

Christina Noren: Yeah, so I'm interested in that, and you're being very modest. You know, you were a Fulbright Scholar in terms of one of the things that – you know, people hear terms like Fulbright Scholar or Rhodes Scholar and we kind of act like it's not important; we don't actually know what it means. So I'd love to understand what being a Fulbright Scholar entails and then I'd also – you know, I'm assuming that that first role was product management in some form at Khan Academy, and we have a lot of software developers on this show. I'm also a product person that's not a software developer, although I've dabbled. So I'm curious about what that experience of turning from an education person into an ed tech person and a product person is about.

Natalie Rothfels: Thanks for tooting my horn. Being a Fulbright Scholar in every country is a little bit different. In my case I was doing a teaching fellowship, so I was working in a classroom and teaching primarily English, although halfway through I started teaching Spanish to university students. They also happened to be university students who were going to become doctors, but in order to enter their program that they had been accepted to needed to do kind of a year of remedial English. So that was its own interesting experience of sort of the gatekeeping that sometimes happens in educational settings and how engaged students are in certain parts of their academic learning just to get to the next phase.

I was also doing some research there about a dying language group called Ladino, and I didn't hint at this but I studied linguistics of all things in college, which is a winding road to where I am right now, but I've always been interested in how language forms community, how it is both the greatest cause and sort of connective tissue between people but it is also a massively destructive force that causes wars, right? And so I was researching this dying language group on the coast of Turkey called Ladino, which has some roots in both Hebrew and Spanish from a group that got kicked out of Spain in the 15th century, so that was kind of my academic nerd side that I spent most of my time teaching.

Christina Noren: Not a lot of distance between being a linguistics nerd and being a computer software nerd, and Paul and I have had many conversations over the years on this. Paul, you did that map of the antecedents of various software languages for Wired years ago, remember?

Paul Boutin: I will jump in and say that compared computer languages. Human languages are far, far more complex and malleable and varied and ever-evolving.

Natalie Rothfels: So my partner also happens to be a linguistics major, so a decent amount of our conversations end up in some rabbit hole of I wonder why we say that thing that we just said or, oh, isn't it funny that the difference between this and this is some obscure thing?

Christina Noren: And then moving onto Khan Academy, so was that a product role that you joined?

Natalie Rothfels: Yes. So when I joined Khan Academy I joined actually in a partnership role, and that partnership role was to find and work with partners in other countries outside the US to develop Khan Academy in other languages. Khan Academy creates their own content, and so it was a real challenge to figure out the sort of strategy and operations behind how do we build out thousands of videos and questions for students outside the US who may be taught in different ways with different curricula in their native languages? So I joined on the partnership side and very quickly I kind of realized and advocated for morphing that into a product role because so much of internationalization is not just translation or getting content in other languages. It's also about what is the right context for that content in the places where you're hoping to deploy it, so it shifted into a product role and that was my first foray into product.

I had no idea what product management was. It's not like I dreamt of being in this role my whole childhood, and still today it's an ever-evolving role that I think is really different in every company and I continue to learn more and more about in my current role.

Christina Noren: As an accidental head of product myself, like I always kind of say that product is everything and at any given time you're doing 10 percent of it and it's going to be a different 10 percent, and the art is to know which 10 percent is important to do at any given time.

Natalie Rothfels: Right, right. I like that.

Christina Noren: So fast forward to Quizlet. What did you join Quizlet to do and then how has that worked out especially through the last five months of change?

Natalie Rothfels: I joined for a couple reasons. One, I had worked at a nonprofit, Khan Academy is a nonprofit, and on a personal side I really wanted to get exposure and understand what is this whole for-profit model and how does it work in the education realm, which kind of more naïve me would have thought like this feels unjust, you know? Education is a public human right, and I absolutely still believe that, and there are lots of problems to be solved in it. So I was really curious about the business side of it, and that was an angle that I was hoping to learn more about and absolutely have since joining the team at Quizlet. The other side was I was interested in a user-generated content platform that was moving – that had solved a core problem of learning really well.

So Quizlet started, bread and butter, kind of digital flashcards combined with using learning science to figure out how you should progress and move through those flashcards to be most effective at learning it. So it solved the core problem of help me recall this information that I may need to memorize really well, and that is a lot of primary education for many students, right? You do need to know some foundational things that you can recall so that then you can recognize them in context but then move kind of up the taxonomy of learning and understand it, apply it in novel context, create things from it. So I was secondarily really interested in how do you take a core problem like digital flashcards for recall, memorization, and start expanding that to help other types of learning or other problems that students have?

So as an example of something we've done in the last couple of months, and this wasn't my team but just Quizlet in general, we built something that we call smart grading. So imagine creating a flashcard where you have on one side a picture of a strawberry and on the other side the word strawberry. Maybe you're an English language learner. That's a pretty simple kind of format of data to store and to represent and to ask questions about. You can just show the picture and then someone can input the text or audio describing it.

But in reality, as a user-generated content platform, we don't get that simple of data, right? We get maybe a giant diagram and then a paragraph of text. And so as a platform one of our goals is to take all of this rich information that a user may give and prompt them with really great questions that may be a recall question or it may be something that is diving more into your understanding of the content about how strawberries need to grow and what environment they have and what are all the processes from going from nothing into a full strawberry to harvest? I don't know, I'm making this up. But we want to be able to ask those difficult questions and we also want to be able to grade or provide feedback on what students are saying to those prompts.

Often you might answer something that is the general gist of things – a strawberry goes from a seed and then it gets germinated and then – I wish I actually knew the answer to this question I just came up with – but nonetheless, we want to be able to understand the semantics of it, right? Smart grading, this cool feature that says whatever you input for your answer to the question we ask you we're going to make sure we kind of understand what did you really mean by that and then grade you based on that. And I think that's an interesting application of technology plus learning science plus student input that helps go from just memorization – this is a strawberry – to deeper understanding. So that's kind of the second reason I joined Quizlet and was really intrigued with what could be possible with the scale and types of data they had.

Christina Noren: So let me understand this a little bit better. So the user-generated content, who is uploading the content and questions? Who is creating quizzes? I assume that's the unit of description. Who's creating quizzes on Quizlet? Let's just start there.

Natalie Rothfels: Sure. Yeah, we call them sets, but quizzes is a great descriptor of them as well. Students are inputting them. So a high school student, I may want to read through it. Where I might have jotted down notes on physical flashcards before I might create those on Quizlet now. Teachers are certainly creating them and they have sort of different functionality that they can use, like adding multiple choice options to their questions so that we provide better distractors that are harder.

So teachers and students both, and then of course not everyone creates content. One of the amazing things about the platform is that we've got probably at this point over 500 million sets of Quizlet, and Spanish I is pretty similar across every school and every country that's learning that, and so students don't necessarily need to make their own content. We already have a ton of content on the platform that may be relevant to them, so they may find something through searching for it on Google or on Quizlet and then they can kind of customize or modify it as they see fit.

Christina Noren: So then what you're describing in terms of the smart grading is it seems like, okay, someone, whether it's a student or a teacher, whoever, creates a set and then you're adding questions to it and you're adding semantics as to how to interpret longform answers is what I'm hearing.

Natalie Rothfels: That's right. That's just an example of how we can use software in novel ways that don't require someone to put in more information or create that quiz for themselves, but we can provide smart ways of prompting and responding and evaluating against those prompts.

Christina Noren: So the devious part of my mind is also thinking about things like auto text generators that are trying to be auto journalists in various ways, and I can imagine that a clever student deploying an AI to answer the questions generated from your AI, but I will try not to go there. [Laughs] So let's come back to just – so your role today, like what is day-to-day, what you're focused on personally, and what have you and your immediate team been doing the last five months that's different that's specific?

Natalie Rothfels: Absolutely. So I work on a new product line at Quizlet that we call the Marketplace, and it's actually one of the reasons I joined Quizlet. We have seen that there is a ton of content on the platform as we just talked about, there's a ton of people creating content and looking for it, but we also are not the arbiters of that content; we don't make it, we aren't content experts in house. And so what my team has been building is what we call prep packs, which are premium content in partnership with third parties who are really good at content, which isn't our core competency. So they may be publishers or small/medium business, in some cases individual teachers themselves, and with a particular focus on certifications and specializations for standardized exams – high stakes, game-changing, often career-evolving exams.

So if I want to become an EMT or a paramedic I need to take an exam for that, right? If I want to be an IT professional I may need to take the CompTIA certification for that. And so to your question about what's happening in the last five months I would actually zoom that out over what's happening over the last 10 years, which is they're slightly different situations, but what we're globally seeing is that the growth of certifications, people wanting to quickly learn something that can enter them into a new job opportunity or advance their career in some way or is a gatekeeping exam for them to say be a nurse, that is a growing space and there's sometimes not great study tools available for learning in those spaces.

On a day-to-day basis my team is partnering with those great content providers in conjunction with the amazing study tool experience that Quizlet has today and is our bread and butter to provide great premium content on our platform, and what that looks like is imagine you're studying for let's say the GRE, right? You've got lots of different content that you have to get through. You may need to remediate on content that you didn't learn before, you've since forgotten the first time you were introduced to it, you've got to in some cases figure out how to schedule your studying, stay on track, understand your progress, how well do I know this information, am I on track to where I want to get, and our goal is to really make these premium packs serve that use case so that students can have a much cheaper but also much more effective time studying for these exams.

So that's what I've been focused on with my team over the last couple of years and definitely seeing some uptick on that as the economy is changing in the last couple of months and people are out of jobs and looking for new paths to take.

Christina Noren: Yeah. It's fascinating to me because it seems like we – you know, it is a 10-year just accelerating right now, but it seems like four-year universities have been turning themselves into vocational schools, but they're not particularly efficient vocational schools, and the economy is going through such major shifts and particularly right now and people do have to retrain themselves for different jobs. I mean part of the whole theme of this podcast Software Agents is that things are becoming software that weren't software before, which means there's software development jobs, there's SRE jobs, there's product management jobs, there's jobs that more so than robots these are the jobs that are displacing what people are doing hands on in physical spaces right now, and we do have this vast need for an effective way to allow the workforce to retrain itself every few years.

Natalie Rothfels: Yeah. That's absolutely right, and I think one thing where I haven't yet called out is access, right? I think a big question that we're all wrestling with societally is who really has access to these materials and these tools and who doesn't and how does that expand the issues of equity that are so clear and obvious and this pandemic has now made even clearer and more obvious? And so I think that's another challenge that we've been wrestling with at Quizlet, is really thinking about who is this for and who is it not for that it should be for? You know, we often cite internally that the average college student now is very different from what you might think it looked like or some folks may have in their head.

It is not an 18-year-old going to a four-year university. It is a 25- or 26-year-old who is going to a community college and has a full-time job and has one or two kids and is juggling all of that at once, and now we've added a pandemic on top of it, right? And so I think something so critical that all companies, but education companies and those working at the intersection of access and resources need to think about, is who has access to this and what gaps are we widening that we need to be very mindful of as we design these solutions?

Paul Boutin: Natalie, can you talk about what you have to do to work with people in education who might not see the value or understand how to implement what you have to offer?

Natalie Rothfels: This question is an interesting one to me because it almost implies that software developers know the right solution for the people implementing their software or needing to use that software, and I think to some degree, hopefully if we're doing our jobs well, we've identified correctly what are the big problems that people are facing and how can we build solutions for them that help? But I also think that's somewhat of a faulty assumption, because the reality is that we aren't teachers at Quizlet, we aren't the people on the ground in classrooms every single day with deep understanding of the inequity issues that face those populations or those needs, and so I think while we do absolutely want to work with teachers and students in using Quizlet and making it super accessible and friendly for everyone who wants to use it, I also think it's important for us as software developers and companies to remain really mindful of the practices that already exist within the spaces we anticipate our software to be used and really focusing on those folks as the experts.

To give you a less vague kind of pontification on that, last week I was in a great – I participated – I was an audience member of a panel about how Bayview-Hunters Point, which is a set of neighborhoods on the Southside of Chicago – on the Southside of San Francisco – are handling distance and remote learning, and on that call, which I joined out of curiosity from a tech standpoint, nobody said anything at any point about access to online ed tech resources. What the entirety of the conversation revolved around was last mile efforts, about how much of the population of those students don't have access to a computer at home, how much of that population doesn't have access to wifi because the two parks in those neighborhoods are the only parks in San Francisco without wifi, how much of those populations are on free and reduced lunch, and so literally just need food, right? I think it's important that we are in the settings that our users are, students and teachers, but that we don't think that we know exactly the reality of how they're going to use that product in their content.

Paul Boutin: That was stellar. And our longtime friend who is a teacher in the San Francisco school system would have said all those things you just said. That was great.

Natalie Rothfels: [Laughs]

Paul Boutin: Just do it online. That assumes you have a computer and a network connection.

Natalie Rothfels: Part of what struck me from that conversation was how great – let's deploy computers to every home. What if that's the only device in the home of multiple students, their cousins, their intergenerational family? Who gets preference? The parent who needs it for work or the child – one of the five children who need it for school? What are we going to do about IEPs, individual education plans that students – you know, we're legally required to support our students who have those plans.

How does distance learning support that in an underfunded educational system? So there's all these kind of systematic issues that really have nothing to do with the software that we're developing for people but I think are so important that we stay grounded in because that is the reality that many students in our country and in our world are in.

Christina Noren: One last line of questioning myself, Paul, and then I'll let us wrap up. So day to day, how does your team work? How do you work as a product person with your engineers, how are you reacting to new ideas and inputs, and how agile and adaptable is the way that your team works?

Natalie Rothfels: We really tactically work in sprints. We try to be super agile. We also do quarterly plans and we have many more teams working at Quizlet now than three years ago when I started. We were one or two PMs at that point and now we've got a 10-person product team and hundreds of engineers, so we can do more and it means we have to coordinate more and it means as we learn more we have to adjust. I mean we're constantly trying to take a step back and say is this still the most important thing that we should be working on as an organization?

We are also responsible right now for making sure that our product stays up for so many people who rely on it for their studies. We are serving more than a billion answers to questions a week on Quizlet. It's a vast number of students who are using it, and so right now, especially in this prime back-to-school time, we need to be responsive to making sure the site stays up and making sure it's performant and making sure it's available across platforms and making sure that the new features we've rolled out that are relevant for back to school and distance learning are accessible as much as possible and available all the time. So we are trying our best to be both forward-thinking and non-reactive while also being nimble and adaptive, and that's always a challenge.

Christina Noren: Do you use continuous integration and continuous delivery tools and how frequently do you release?

Natalie Rothfels: We do. We have a pretty robust way of doing this and I am not the right person to describe it, but just yesterday we have kind of a change log channel in Slack and there were 15 different releases of new things that went out yesterday across all these different teams, so it's something we rely on all the time. On top of that we are a software company, so the complexity of this is about scale and supporting new features across platforms. We luckily don't have to manage quarterly deployments for hardware going out to people, so that makes our life somewhat easier, but it's definitely part of our process.

Christina Noren: So Paul, I don't have any more questions for Natalie unless you wanted to take this to another hour, which we could do very easily, but my takeaway is that Quizlet is something that was a natural direction of the world to go in terms of facilitating more self-guided and self-directed online learning, and that's been a huge topic of how we're adapting to this time. So it's another case of where the right software was already built and being used and has gone from nice, great to have to critical. Paul, what's your takeaway?

Paul Boutin: Well, I heard as a previous guest said that bringing solutions to people starts with understanding that they understand the problem not you. How can you help them solve it in ways that maybe they're not familiar with? And the other thing I heard is that when you try to bring something to an underserved group you find things that you didn't think of, right? Let's give everyone a computer. Well now you've given one family a computer so they can fight over it, which actually happens.

That being aware of that is part of being part of the solution. Also, 15 rollouts in one day; that is very different from the literal old school of getting new rollouts every few years.

Natalie Rothfels: Yes, it is an ongoing process, and trying to make sure that at any given moment we know exactly how the product works is a fun challenge across the team too since the more students we get to serve, the more teachers we get to serve, so too the number of use cases we're trying to support evolve and that's a fun matrix to try to solve against while also keeping things as simple as possible.

Christina Noren: Natalie, thanks so much for spending the time with us today. You know, it's really fun to peer into each of these different corners of the world with someone who's deeply enmeshed in it, so thanks for your time, thanks for your thoughts, and thanks for the work you're doing.

Natalie Rothfels: Thanks for having me. It was fun.

Christina Noren

Follow Christina Noren on Twitter and Linkedin.