Method To The Madness
Interview of theoretical scientist and Chief Data Scientist for GILD in San Francisco on how to use big data techniques to design more democratic and merit based hiring practices.
Speaker 1: Method to the madness is next. You're listening to method to the madness, a biweekly program celebrating Bay area innovators. I'm Lisa Kiefer and today I'm interviewing Dr Vivienne Ming, chief scientist at Guild, a talent acquisition tech company in San Francisco. I read about you in New York Times, so tell me what you do at guild is a company whose goal is to bring meritocracy back to tech hiring. We have customers that [00:00:30] are looking for programmers. All the biggest tech companies you can like Google and like Google, Facebook, Microsoft Tho branching now, people that you wouldn't think of like Nike, some banks and others. It just, it's so pervasive. So much of what we do is based on some kind of programming. Every company has somebody that they need to hire in this space. The founders short sigh and Luke Obama soar, decided they wanted to create a company that could go [00:01:00] beyond qualities in a resume that they know.
Speaker 1: We could just look at. The easiest way to get a job at Google is be good, but go to Stanford and know people that are already working. If you don't fit those two qualities, it's not a knock on Google. They get so many resumes that in some sense, what else are they going to do? So as chief scientist, what I do is come up with algorithms to go beyond that. How many variables are you looking at and could you talk about what some of those might be? Currently we're looking at 50,000 [00:01:30] different features as we call them about a person that boils down to something on that order of a hundred independent dimensions. Each of those dimensions is saying something unique about the people that we're looking at or each of them weighed differently. Each of them are weighed differently and one of the cool things we do is that they're weighed slightly differently for slightly different companies and where we're in the process of developing and advancing these algorithms all the time.
Speaker 1: So the number of features increases. The weighting on these factors increases. [00:02:00] We can go to these companies. They say to us, I'm looking for a Java developer in Boston and we return a list ordered by how good we predict people to be of the Best Java developers in Boston. Okay. Now talk about some of these variables on the higher east side. Some of these variables have to do with how people express themselves, not generically, but specifically related to the profession that we're recommending them for. Some of these are very simple things. We just look at [00:02:30] what social sites they spend their times on. That would give us a little nudge in one direction or another in our estimate of how good you are. The way you describe yourself on your resume on linkedin. Actually don't look that much at Facebook because Facebook really strongly represents what you want other people to think about you rather than who you actually are to strange quality.
Speaker 1: There's a lot of information there. I was going to ask you about that. I think all of these sites are pretty easily gamed. Say a company's looking for someone to be a c, so c [00:03:00] is a fairly low level programming language. It's used by people to build really fundamental pieces, a very fast processing. There's also a language called C plus. Plus. It's very similar to c in its application, but it has a different and a pretty fundamental one and how it's structured. You will very commonly see on resumes that someone is proficient in c slash c plus. Plus if they say that our algorithm predicts that they are not a good c or c [00:03:30] plus plus programmer. Why? Because these are different languages so you are professional programmers. C was your space for doing things. Even if you happen to know c plus plus, that's not how you describe yourself.
Speaker 1: At least as we look across the 4 million profiles in our database, that is not how the best c programmers or c plus plus programmers described themselves. So your algorithms sounds like they're going to be constantly changing, but the more information you get into this, and in fact we built it with what we call temporal discounting. So over time [00:04:00] it tends to ignore things that happened a long time ago and really focuses on right now. So that allows us to have a bit of a memory in a sense. I can say something like what I just said because I know our algorithm will adapt if people search to start to try to game it. But at the same time the tech world is so fast moving, it has to adapt. You know, if we recommend someone as a highly qualified programmer because they use a technology that was popular 10 years ago, then we're probably not doing a service to our customers.
Speaker 1: Are you only servicing [00:04:30] tech companies in the bay area? Certainly, but we certainly service ecommerce companies like Walmart, they have an incredible presence in, in technology. A long before, in fact, a lot of other companies were doing big data. They had huge servers full of everyone's behavior at Walmart, uh, that they were analyzing. Look at our co founder and chief technology officer came up with the original idea of let's look at open source code. So this is code that developers write freely to share [00:05:00] amongst themselves. And this isn't trivial work. Some of the absolute backbone of our technology infrastructure is based on open source code. And this ranges from Linux, which runs a vast amount of computing and web serving and everything around the world to machine learning languages like [inaudible]. It's all just freely given out. Luca came up with the idea, why don't we go there and actually look at the free public code that they put out and evaluate it.
Speaker 1: He wrote this just fantastic system that goes [00:05:30] through and reads their code, reads their contribution cause many of these projects have many people on and we can split that out and evaluate how good they are as a programmer. And so our original system was based on that. Some companies like Google and Facebook actually do open source as part of their internal development for any techies listening. Things like Hadoop and Cassandra have been turned out by Yahoo and by Facebook just freely for the use of the rest of us. But they built it for themselves [00:06:00] internally. That's awesome. But many, many tech companies, particularly a lot of these big server-based companies like IBM, they don't do that. And so there's a whole army of people working there that we traditionally don't have insight into. We have hundreds of thousands of people that we can look at and evaluate.
Speaker 1: There are millions of developers out there. We very roughly estimate about 8 million professional working developers in the world. Yeah, we have a database in the u s Europe, China and large parts of Asian India [00:06:30] of roughly 4 million. I've been amazed and been told by some of our customers that some of their best results have come by looking outside the United States within our database. So we want to take those hundreds of thousands of people that have gone out and done something wonderful and very accurately convey to our customers how good they are. This is k a l x Berkeley 90.7 on your FM dial and streaming on the email@example.com I've been talking a lot about almost surface level [00:07:00] information that we pull out of these sites like you know, did people really like the answers? You Post it on stack overflow and how often was your code on get hub? How often was the code on get hub polled forked as they call it and used by others or followed by others.
Speaker 1: But we can actually get more sophisticated than that. We can literally go in and evaluate the content of what people are saying. I can tell how what kind of person they are in essence and, and I think many of our customers would be interested in us putting out a product that [00:07:30] can actually say this person is a good personality match for you or a good, you know, match in terms of housing and all of these search firms and we're not trying to build something to replace the existing systems per se. Some of them need replacing, they need disruption. Yeah. As they say, disruption. But even starting more simply. A lot of recruiters with titles like technical recruiter are not technical people themselves, but many of them, they get a resume [00:08:00] and it says, we need someone with flask experience. Well, what does flask does? Recruiter doesn't know.
Speaker 1: And it's not because they don't know their job, it's because that's a pretty specific technology. It's a subset of python, which is a subset of interpretive languages and it isn't necessarily their job to know this, although wouldn't it be nice if they did and then they get a resume and that resume says the person works in Django. Well, [00:08:30] little do they know those are highly compatible technologies. That person may be a great candidate, but if they don't see those matching words. Part of our research right now and some exciting potential products to come is based around being able to turn people into instant experts, essentially designing systems that will understand the ontology, the taxonomy of the technology, maybe other worlds as well. Wouldn't it be nice if we could just say, [00:09:00] tell us who you love at your company right now, who is incredible and you say, oh well and Brad and I love Jill and Brad and we just said, Oh care 20 more Jill and brats.
Speaker 1: And you say, oh well these five, not quite what I meant. And we say, oh, thanks here. Another five that fit even better and you can turn that experience into the recruiting experience. It's like you did one interview and we behind the scenes populate the results of your interview [00:09:30] with the ideal candidate or their ideal set of candidates. Your job is to simply go out and do the recruiting. All we care about is whether they fit the job, what you just said you need it. I saw Ray Kurzweil speak here recently and one of the things he was talking about was the ability to know before you even know what you need. Well that's the beauty of what I'm just describing myself. For example, I just recently found out my team for guild. I started the process by trying to scratch down ideas of who would be the right candidate and then we start [00:10:00] the process and we realize, oh, that's not quite right.
Speaker 1: And then we go back and we kind of iterate a little bit and you know, my recruiters looking to match the specific terms on my job description I've written up and it's an ugly and inefficient process and it's inevitably going to miss great candidates, great candidates that don't fit the obvious mold of a great candidate. Like the guy that in the New York Times article, jade jade, okay, he didn't go to college. Jade has this amazing story, no obvious [00:10:30] exceptionalism in high school, no work history that speaks to the corporate world or even the startup world. You wouldn't just not bring them in for an interview. His resume probably wouldn't even get in the door. Why would you ever consider someone like this? Well, you'd consider him because he's an amazing front end developer and he's done amazing work for the US and Luca discovered him using the algorithm look a developed by saying who is the best front end developer in Los Angeles?
Speaker 1: Essentially that was his question. [00:11:00] There was jade with a perfect score right up there, like a guide no one would ever look at, you know, we call them up and of course [inaudible] us while at this tech company in San Francisco has startup. What do, they brought him up for an interview and it clicked and he does great work. You know, as the article says, this is kind of an experiment. I think an experiment, which I can personally say jade is gonna do great things and I love him. It's fun having him in the office. There was a huge discovery for [00:11:30] us. People that would otherwise get ignored, have a legitimate shot at jobs. They're qualified for it. In fact, my research, as for every one of those standouts, there are hundred times as many people that are just as qualified. The tragedy isn't that the credential people are getting the jobs, they deserve it.
Speaker 1: The tragedy is all of those other people being left behind. And we have the opportunity now to look at this here at 10 people were saying they're all equally qualified. You've got the money and the opportunity and you want certainty. [00:12:00] Okay, hire the Stanford candidate, the MIT candidate, the cal tech candidate, but if you want somebody good and you don't have that money or maybe you've lost out to Facebook and Google, not everybody can throw $1 million just to get someone to come work for them. There's a real market distortion. A small number of people are being highly overvalued and it is scorched earth and silicon valley trying to find those proven developers. There are a lot of people out there. The question is how do you find them? How do you validate them? Facebook and Google are testing [00:12:30] our system, not because they need us to find candidates because they want to find the candidates.
Speaker 1: They can't find other ones to use their language. They want to find diversity, fully qualified, equally qualified candidates. Our system does not over promote anybody. You have to make it there on your merit. Open source is a wonderful thing to do for the world, but it's also a demonstration of who you are and what you can do. Even small projects, we use those and believe me, recruiters look there. Also, if you're just tuning in, you're listening to method to the madness on [00:13:00] k a l x Berkeley, and today I'm interviewing Dr Vivienne Ming, chief scientist at Gild it talent acquisition tech company in San Francisco. I have a thread that runs through all of my work, so I'm a visiting scholar here at Berkeley at the Redwood Center for theoretical neuroscience. I have a company that I co founded with my wife and a former student of mine called socos where we do cognitive modeling of students for educational technology and I even dabble around with things like Google glass that I'm wearing [00:13:30] right now and working in modeling diabetes, which applies to my son who has diabetes.
Speaker 1: Across all of that. What's important to me is maximizing human potential at the Redwood Center. I'm interested in neuroprosthetics particularly what we call cognitive neuroprosthetics. People that have Alzheimer's, that have hearing loss, that have decreased working memory spans that have autism. Imagine what we can do with the technology that's coming up to compensate for these [00:14:00] Google glass. For those of you who don't know is that sort of experimental development of project that Google's actually put out in the wild now. So I'm wearing a pair right now and they don't look too bad either. People know and they're going to look pretty cool. I like them. It's voice activated and so I can turn it down with the head nod and say, okay glass, take a picture and there we go. I just took a picture so I can essentially see Google search results. I can see videos, get directions.
Speaker 1: Imagine I'd put this on an autistic child. [00:14:30] I've done previous research in automatic facial expression recognition. Imagine the video camera is watching the expressions of the person I'm talking to, processing it back on a server and then in the little pop up I'm telling the child what, what emotion that person is feeling so they have a chance to get a real time feedback on their interactions. Imagine people in Boston had been wearing glass, the explosions go off and there's 20 people say, okay glass virtual EMT and they are alive, connected [00:15:00] with an emergency room doctor working at desk. And the doctor can see what they see through the camera. I can hear the doctor in a Mike that goes into my ear and in a heads up, I can see them talking to me. I can read your heart rate right off the camera just from subtle changes in your skin color and body temperature and things like that, and suddenly those 20 people went from being shoppers and runners to being first responders.
Speaker 1: The idea of what you can do with people through neuroprosthetics as I call them and now [00:15:30] it's the augmenting cognition is is just amazing. Retinal implants, motor prosthetics, people learning to move quadriplegics and stroke victims and strikes with people that haven't moved parts of their body in years and years. This really laid the groundwork. It was what had got me into graduate school. It's what drives my academic research still. When I was given a chance to think about, for example, cognitive modeling of students, I wanted [00:16:00] the opportunity to go out and bring that onto the world instead of being an academic project, which is incredibly valuable. My wife and I founded a company so that we could access it soco. So that's so close. I was working at the time as a research scientist at UC Berkeley. Uh, my wife was a lecturer. She studies the learning sciences, which is sort of cognitive psychology for education.
Speaker 1: And I had a student at the time, the most amazing guy, he's at the Ed school at Stanford now. His name is [inaudible]. We [00:16:30] decided we wanted to start a company where we could do something amazing, which was figure out whether students understood what they were talking about in their own free form discussions, talking to other students, interacting with instructors, sending emails, doing homework. So helping teachers know where they're not reaching students. Exactly, but to do it without imposing anything on them. There's a lot of buzz around Ed Tech Con Academy and you know a lot of [00:17:00] work by the gates foundation. Companies like dreambox and Carnegie learning and others putting out really amazing technology. But one aspect of most of that technology is this is the learning experience. We have decided a curriculum for you, if you want to adopt this for your classroom, this is what the experience will be and we'll need to retrain your teachers and we'll bring the computers into the classroom and the kids will solve math games.
Speaker 1: That took you know, years to really optimize [00:17:30] and get just right and there are proven effective, at least in the lab. They have some challenges in the wild though teachers don't buy in. The curriculum isn't quite adopted correctly. Hard to track exactly what students are doing. Wouldn't it be better and sort of more responsible for us as technologists to say, teachers, curriculum developers, you're the experts. Go explore and educate the way you want to. Just share with us everything that that experience producers and it will be our heart job [00:18:00] to make meaning out of it. We looked at an introduction to biology class and an MBA class in economics and we simply looked at their online discussions. What we found was one, we could learn biology and economics just by listening to the students. We didn't need to model a textbook ahead of time or bring an expert in to build our system instead of an expert system about biology.
Speaker 1: We had an expert system about how students thought about [00:18:30] biology or what they knew or what they knew, so it included the right and the wrong and it included it with nuance and then when we took in a new group of students with new instructors, we found in week one we could predict what grade they get in the class. Again, just from their freeform discussions, not looking at homeworks or essays or final exams. By the end of the class we had an extremely tight understanding of what they knew and how they would perform in the class. The final grade they would get and the vision is wouldn't it be great? [00:19:00] Then back in week one, if we could say to the student, the learner, to the instructor, we predict these students share misconcept and historically looking at other students, we found that these interventions like a reading or lecture or homework experience were effective in moving students from this misconcept to this more normative concept.
Speaker 1: The teacher teaches the class the way they want and the way they should because they know what their kids aren't getting. They are the expert. And we simply [00:19:30] essentially in real time, give them feedback on which students are getting it and which aren't and effective ways they might go back to the students that aren't. And this is in practice right now somewhere. Um, we've published papers on it. We're in touch with a couple of prominent educational technologies, companies that want to use our system as the intelligence behind their amazing products. So we're gonna make people a lot smarter. That's our goal. And then there's going to be a lot more competition for all those great jobs you're finding [00:20:00] too. Well, again, so we're looking at maximizing human potential and the ability of our system is to identify the unique understanding of a given student and really try and move them in the most positive direction we can.
Speaker 1: We are incredibly passionate about the ability to understand student cognition and really create ais that are just personal tutors that will go with students with the rest of their lives. Here's our big thing for soft costs and all standardized [00:20:30] testing. I, I get the sense that your life is definitely informing your work. Everyone always thought I would be really good at school. My Mom, my dad being the sort of crazy geemer that he was just was convinced you are again, you're gonna get a Nobel prize someday. I know he was incredibly successful. Had he got a bronze medal in Vietnam, right? Did He, I mean, he was like an amazing helicopter surgeon I would with him. So he grew up on a farm, five kids and his graduating class, I think he [00:21:00] got full scholarships. He was an amazing man. As a, as a doctor in the community, specifically at gastroenterologists, you know, treating all the patients that come into his door.
Speaker 1: He instilled in me the belief that you should leave a life of substance. And it's why I choose to do the work that I do. But my mother's a teacher out of Kansas as well, worked for decades and a, and a great teachers, Sixth Grade Public School, Salinas, California. She is an amazing woman. They expected [00:21:30] things of me despite the fact I typically was failing all of my classes through high school. Through my first years of college, I was very unhappy growing up. The only way my father agreed to send me to this private high school, Robert Louis Stevenson, is if I played football. But he had these fears. This is back, you know, early eighties you know, some froofy private school might turn my son gay. Little did he know that it was that very experience [00:22:00] that totally clarified the world for me and the world being, I never understood the other pies.
Speaker 1: Their behavior, casual sexual jokes made no sense to me. I'll be honest, I thought everyone was an idiot but me, and then I understood I was the one that was different. When did you come to that realization? This was when I was 12 the understanding didn't change anything and in some ways it sort of made it worse because okay, I was a boy and I didn't want it. B, what good does that do? That just makes life harder. So I got through high [00:22:30] school is the best way of describing it. I loved academia. I was planning on being a doctor, like 85% of the undergrads at UC San Diego. It was just basically a big biotech school, and I showed up there and now no one was even looking over my shoulder and I wasn't doing the homework and then I wasn't going to class. And then I wasn't even bothering sharp with the final gone.
Speaker 1: So you had not confronted either your mother father at this point with how you felt. So you know, now we're into my twenties by that point I considered the idea of gender transition, [00:23:00] but I was so isolated and so alone and no support. So I'm starting to learn a little bit, but I'm not part of any community and I'm thinking, how am I going to keep going? Being unhappy. I completely stumbled into a job without looking for it. Running an abalone farm in Santa Cruz, California, the economy in Japan crops. So there our main customer base, and now they're not buying our Sushi anymore when the end came, because it was inevitable. I had saved up a little bit of money and I thought, why don't I just go back to school and try [00:23:30] and do something substantial. So you had not finished your undergraduate? Not finished.
Speaker 1: My Undergrad, I'd been, I think I'd been there three years. What degree could I finish in a single year? I literally flipped a coin between economics and cognitive science, cognitive science one I thought, okay, I'm going to be a neuroscientist. I went there and started taking classes and they were just like ridiculously easy. I was getting A's and a pluses and everything compared to having worked at this abalone farm where you know, the world was falling apart every single day [00:24:00] and my, my love of research and academia finally had fertile ground where I actually got successful feedback in one class. The professor came back and said, I've got a research project, which I like to work on it and that eventually led me into this field of theoretical neuroscience. I applied to Grad schools and I was still presenting male at that time. I had a very deep voice and a presence and I was getting a lot of the benefit of the doubt.
Speaker 1: You [00:24:30] know, I'd come into psychology departments and talk sophisticated mathematical ideas about cognition with that presence and people would start nodding their heads and saying, you know, would you want to come join our lab? Of course I cherish those opportunities, but I always kind of felt like a fraud, do really know exactly. And I gained it all the way to Carnegie Mellon, which is an amazing place. And I worked there with several people. Jay McClelland, who's Jay is now at Stanford, Mike Wiki now at case western and [00:25:00] Laurie halt, who is still at CMU. I just loved working with all of them, but I was still fundamentally unhappy. I was having all of the success fighting, you know, our, my work with Mike was published in nature and I had chances to get up in front of hundreds of people at major conferences and talk about our research and feel good for five minutes and then it was gone.
Speaker 1: And then I'm out. Norma. Um, we were together in the psychology department at Carnegie Mellon early on in our weird courtship. [00:25:30] She taught me deep, dark secrets about herself. And all I said was, I've got a pretty big secret to maybe I'll tell you to you someday. Four years later, our final year at Carnegie Mellon finish finishing our dissertations. It was my birthday. We were together. We were actually engaged at that point. I'm just going to be the best husband that I can and I'm successful at work. I have someone that makes me happy. So many people don't have either of those things, much less, both of them. I don't know how it came about, but I was invited to be in an experiment to [00:26:00] look at the effects of NZ Alytics. So anxiety reducing medicines on heart health. And it was a blind double-blind. No one knew what medicine they were getting.
Speaker 1: So I was taking something turned out the medicine they were testing is called Celexa. But I didn't know if I was taking any. And in retrospect it was so obvious that like the change in behavior and so forth. But turned out I was, I learned after the fact I was in the treatment group. Why was that so fascinating? Because there was in the midst of this and looking back, I realized, wow, I wasn't shouting at any people. I'm, I'm, I [00:26:30] was like a notorious angry driver. And I said, wow, I haven't like shouted at anyone in the car in like months in the midst of this for whatever reason. I taught in a moment total freedom to just share that with Norma. My big, deep, dark secret is I wish I were a woman. You weren't married yet, right? We weren't married though. We weren't engaged.
Speaker 1: Um, we stayed up that night talking and, and we talked for about a week. That was the start of my transition. Completely unplanned, completely unexpected, but with her full support. But we loved each other. Her parents still [00:27:00] really struggled. How about yours? What makes me most happy is that before my father passed away, he was back to bragging about me again. He had some struggles and he had troubles with pronouns. I mean our parents, I will single out my mother, she had like three days of tears and then a light switch and she was like, all right, we've got to go get you a new wardrobe yet you need professional outfits. I mean, she just has been amazing from that moment on. My siblings have been incredibly supportive. Enormous siblings have been incredibly supportive. My friends, [00:27:30] friends that I most feared coming out to it, their response was amazing.
Speaker 1: Normally this decision is a decision to start a new life, not because you want to because your family leaves you and your friends won't talk to you and your career is offering. Most importantly to me, Norma is still enormous source of happiness. Our children aren't enormous sources. Do you have a boy and a girl? Right? I have a boy and a girl. Um, I'm their mommy. I'm also their donor. Did you have the foresight to a banked uh, ahead of time? Very [00:28:00] good. That tells me a lot about how this idea of merit and bias, like you say, how people treated you as a man and you were gaming that. So that has informed a lot of your algorithmic work at gala. I feel like a secret spy having seen all of this, they're a man's eyes and that's such a good thing to do to try to eliminate bias in the hiring practices of the workforce, whatever they might be.
Speaker 1: At Guild, I had the opportunity to work in a company [00:28:30] whose motto is meritocracy. We want to give everyone a legitimate shot at what they're qualified so I could take my expertise there and apply it. Again, back to my life's goal of empowerment and maximizing human potential, and so guilt really has become an amazing platform for that. I bet a lot of our listeners are to want to get ahold of you or talk to you or maybe ask you a question. Do you have a website that you would recommend they look at both for guild questions, but also LGBT [00:29:00] questions are an absent that. If you look us firstname.lastname@example.org you can see the kind of work that we do and you can learn a little bit about me and the founders there and our work in meritocracy. Our education email@example.com, s o c o s.
Speaker 1: Dot. Emmy at the Redwood Center for theoretical neuroscience here at UC Berkeley. It's at redwood.berkeley.edu [00:29:30] you can see all of the amazing research we do. Very Geeky. You'll love it. Finally, there's my own website. If you just want to reach out to me personally and maybe on LGBT issues or anything like that, you can find firstname.lastname@example.org I am not much of a social networker, but I love to sit down and talk with people. I told a group of students here from Stanford and cow yesterday, learn to do something of value so that you'll have some tools [00:30:00] for the rest of your life. Learn engineering or learn the practical skills of putting words on a page, whatever it is, but learn something tangible that other people will value. Commit fully to that amazing thing you're doing right now. You've got a whole life ahead of you to do more amazing things. That's kind of how I personally have embraced the very weird and and incredibly fortuitous life. I've had the chance to have an amazing life of Dr Vivian [00:30:30] Maine. Thank you for being on this program. I've really enjoyed it. It was a real pleasure. If you have any questions or comments, go to our website method to the madness.org that's all one word. So you in two weeks at the same time.