Share

cover art for Jitendra Malik on the sensorimotor road to artificial intelligence

Berkeley Talks

Jitendra Malik on the sensorimotor road to artificial intelligence

Ep. 164

Jitendra Malik, a professor of electrical engineering and computer sciences at UC Berkeley, gives the 2023 Martin Meyerson Berkeley Faculty Research Lecture called, "The sensorimotor road to artificial intelligence."

"It's my pleasure to talk on this very, very hot topic today," Malik begins. "But I'm going to talk about natural intelligence first because we can't talk about artificial intelligence without knowing something about the natural variety.

"We could talk about intelligence as having started about 550 million years ago in the Cambrian era, when we had our first multicellular animals that could move about," he continues. "So, these were the first animals that could move, and that gave them an advantage because they could find food in different places. But if you want to move and find food in different places, you need to perceive, you need to know where to go to, which means that you need to have some kind of a vision system or a perception system. And that's why we have this slogan, which is from Gibson, "We see in order to move and we move in order to see."

For a robot to have the ability to navigate specific terrain, like stepping stones or stairs, Malik says, it needs some kind of vision system.

"But how do we train the vision system?" he asks. "We wanted it to learn in the wild. So, here was our intuition: If you think of a robot on stairs, its proprioception, its senses, its joint angles can let it compute the depth of its left leg and right leg and so on. It has that geometry from its joint angles, from its internal state. So, can we use it for training? The idea was the proprioception predicts the depth of every leg and the vision system gets an image. What we asked the vision system to do is to predict what the depth will be 1.5 seconds later.

"That was the idea — that you just shift what signal it will know 1.5 seconds later and use that to do this advanced prediction. So, we have this robot, which is learning day by day. In the first day, it's clumsy. The second day, it goes up further. And then, finally, on the third day, you will see that it ... makes it all the way."

Malik's lecture, which took place on March 20, was the first in a series of public lectures at Berkeley this spring by the world's leading experts on artificial intelligence. Other speakers in the series will include Berkeley Ph.D. recipient John Schulman, a co-founder of OpenAI and the primary architect of ChatGPT; a professor emeritus at MIT and a leading expert in robotics, and four other leading Berkeley AI faculty members who will discuss recent advances in the fields of computer vision, machine learning and robotics.

Listen to the episode and read the transcript on Berkeley News.

UC Berkeley photo.

Music by Blue Dot Sessions.

More episodes

View all episodes

  • 180. What are Berkeley's Latinx Thriving Initiatives?

    52:31
    In Berkeley Talks episode 180, Dania Matos and Fabrizio Mejia, vice chancellor and associate vice chancellor, respectively, for UC Berkeley’s Division of Equity and Inclusion, join Berkeley student Angelica Garcia to discuss the campus’s Latinx Thriving Initiatives (LTI) and how these efforts are supporting Berkeley’s goal of not only becoming a Hispanic Serving Institution (HSI), but also of transforming Berkeley into a Latinx Thriving Institution.“There's a practical standpoint of this that's about becoming a Hispanic Serving Institution,” begins Matos. “That's why you'll hear HSIs a lot, and it's important in that naming and framing. Dr. Gina Garcia is a sort of national expert who talks about this, but for us, it's really taking it beyond that. Because becoming an HSI is about 25% enrollment of undergraduate students, which, by the way, the federal government does not count graduate students, and we care about graduate students here, too.“For us, we're thinking about (how to) build this ecosystem where we are honoring, bringing in more Latinx communities, but also honoring the different ways of knowing and being, which are so counter to the way U.S. higher educational system is done.“Latinx Thriving Initiatives is a multifaceted campuswide effort, but not just campus. It's really thinking and honoring our mission as a public institution, and (asking), ‘How do we center community in that?’ We're not just a community center that people come to or get admitted to and leave from in four years, but a place where they're having an impact and creating that.”“It’s collecting all the work that is happening,” added Mejia, “honoring that historical effort, that ongoing effort, but also asking ourselves, ‘What else? What does it look like to have a Latinx student body, Latinx staff, Latinx faculty that are all thriving from the moment they conceptualize “Is Berkeley a place for me?” to the moment they land on the campus and say, “Where do I see myself in this environment? How do I meet my aspirations? Am I getting the support all the way through?"’Listen to the full conversation in Berkeley Talks episode 180, “What are Berkeley’s Latinx Thriving Initiatives?”This conversation was recorded in March 2023 as the first episode of Berkeley’s Latinx Thriving Initiatives podcast, which explores what it means for Berkeley to become a Latinx Thriving Institution, and the direct impact it’ll have on its Latinx-identifying campus community. It was created in partnership with Ethnic Studies Changemaker, a campus group of students plus a faculty adviser that aims to “amplify the voices and diverse experiences of marginalized communities.”Learn more about Berkeley’s Latinx Thriving Initiatives and watch a video of what these efforts have accomplished over the past year on the Latinx Thriving Initiatives website.Photo by H.G. Villaseñor and I. Torres.Music by Blue Dot Sessions.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).
  • 179. Poet Ishion Hutchinson reads 'The Mud Sermon' and other poems

    41:06
    In Berkeley Talks episode 179, Jamaican poet Ishion Hutchinson reads several poems, including "The Mud Sermon," "The Bicycle Eclogue" and "After the Hurricane." His April reading was part of the UC Berkeley Library’s monthly event Lunch Poems."I take this voyage into poetry very seriously," begins Hutchinson, "and take none of it for granted, because of the weight of history, both growing up in Jamaica and knowing the violent history that comes with that. But also the violence, too, of canon, and seeing that my work as a poet, in part, is to figure out what sort of emancipatory forces I should summon. Luckily, I stand in great shoulders within the Caribbean tradition of many poets and writers that I admire, and envy, and wish they hadn't been born. Don't tell them that. This isn't recorded, of course."Here’s “A Mud Sermon,” one of the poems Hutchinson read during the event:They shovelled the long trenches day and night.Frostbitten mud. Shellshock mud. Dungheap mud. Imperial mud.Venereal mud. Malaria mud. Hun bait mud. Mating mud.1655 mud: white flashes of sharks. Golgotha mud. Chilblain mud.Caliban mud. Cannibal mud. Ha ha ha mud. Amnesia mud.Drapetomania mud. Lice mud. Pyrexia mud. Exposure mud. Aphasia mud.No-man’s-land’s-Everyman’s mud. And the smoking flax mud.Dysentery mud. Septic sore mud. Hog pen mud. Nephritis mud.Constipated mud. Faith mud. Sandfly fever mud. Rat mud.Sheol mud. Ir-ha-cheres mud. Ague mud. Asquith mud. Parade mud.Scabies mud. Mumps mud. Memra mud. Pneumonia mud.Mene Mene Tekel Upharsin mud. Civil war mud.And darkness and worms will be their dwelling-place mud.Yaws mud. Gog mud. Magog mud. God mud.Canaan the unseen, as promised, saw mud.They resurrected new counter-kingdoms,by the arbitrament of the sword mud.Ishion Hutchinson was born in Port Antonio, Jamaica. He is the author of two poetry collections: Far District and House of Lords and Commons. He is the recipient of the National Book Critics Circle Award for Poetry, a Guggenheim Fellowship, the Joseph Brodsky Rome Prize, the Whiting Writers Award, the PEN/Joyce Osterweil Award, the Windham-Campbell Prize for Poetry and the American Academy of Arts and Letters Award in Literature, among others. He is a contributing editor to the literary journals The Common and Tongue: A Journal of Writing & Art, and teaches in the graduate writing program at Cornell University.Lunch Poems is an ongoing poetry reading series at Berkeley that began in 2014. All readings happen from 12:10 p.m. to 12:50 p.m. on the first Thursday of the month. A new season of Lunch Poems will begin on Oct. 5 with Inuit poet dg nanouk okpik in the Morrison Library.Find upcoming talks on the Lunch Poems website and watch videos of past readings on the Lunch Poems YouTube channel. Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Photo by Neil-Anthony Watson.Music by Blue Dot Sessions.
  • 178. Michael Brown's family on keeping his memory alive

    01:48:27
    In Berkeley Talks episode 178, Rashad Arman Timmons, a fellow at UC Berkeley’s Black Studies Collaboratory, joins in conversation with the family of Michael Brown Jr., whose 2014 killing by police in Ferguson, Missouri, ignited a wave of protests across the country.During the March 8, 2023, discussion, Brown’s father, Michael Brown Sr., his stepmother, Cal Brown, and Timmons consider the enduring significance of Ferguson in the nation’s racial landscape and ponder Black grief as a resource for social transformation.“A note on grief,” begins Timmons. “We grieve because we care. We grieve because we love. And we grieve because we remember. I feel a responsibility to say this, to acknowledge grief for what it truly is: an ethical act of care, a radical act of love and a persistent triumph of memory.“When we grieve the Black dead and dying, we enact an urgent care for them. We profess a vigilant love over them and nurture a commitment to remember them. Christina Sharpe in her beautiful theorizing calls the unison of these practices ‘wake work.’ ‘Wake work,’ she writes, "describes how we attend to physical, social and figurative death, and also to the largeness that is Black life, or Black life insisted from death. Wake work describes how we imagine, defend and care for Black lives always already threatened, in our present or the future, that chattel slavery made possible.”Read more about the event and learn more about the Black Studies Collaboratory.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Music by Blue Dot Sessions.
  • 177. Oppenheimer's Berkeley years

    01:27:34
    In Berkeley Talks episode 177, a panel of scholars discusses theoretical physicist J. Robert Oppenheimer and how his years at UC Berkeley shaped him, and how he shaped the university.Oppenheimer, the subject of Christopher Nolan’s summer 2023 film Oppenheimer, came to Berkeley in 1929 as an assistant professor and over the next dozen years established one of the greatest schools of theoretical physics. He went on to direct the Manhattan Project’s Los Alamos Laboratory during World War II, during which the first nuclear weapons were developed. He’s often referred to as “the father of the atomic bomb.”“Exceptional students and postdocs flocked here to Berkeley to work with him,” began Cathryn Carson, a Berkeley professor of history and a specialist in the history of 20th century physics, who moderated the July 28 discussion at Berkeley.“As we’ll hear today,” she continued, “the style of work that Oppenheimer unfolded at Berkeley was collaborative, pointed, directed at hard problems, not always successful. His modus operandi, you could say, was, ‘Work hard, play hard.’“He landed in the Bay at a time when much else was in ferment. At the same time that he devoted himself to physics, he got engaged with contemporary left-wing politics. In the Bay Area in the 1930s, that included the fight against fascism in Nazi Germany and Spain and struggles for economic justice and labor in California. The Communist Party was part of that setting, and Oppenheimer immersed himself in the life of the Berkeley faculty, efforts to unionize it and intellectual currents across the university — this broad liberal arts institution that fed his roving mind.”Panelists include:Cathryn Carson, chair and professor of Berkeley’s Department of History, whose research includes nuclear history and the history of 20th century physics. She co-edited a volume of papers about Oppenheimer, Reappraising Oppenheimer: Centennial Studies and Reflections.Mark Chadwick, chief scientist and chief operating officer for weapons physics at Los Alamos National Laboratory, who edited and published a suite of papers on the technical history of the Trinity test.Jon Else, professor emeritus of Berkeley’s Graduate School of Journalism, who created the documentary The Day After Trinity: J. Robert Oppenheimer and the Atomic Bomb.Yasunori Nomura, a Berkeley professor of physics and director of the Berkeley Center for Theoretical Physics.Karl van Bibber, professor of nuclear engineering at Berkeley, who spent 25 years conducting nuclear energy research at Lawrence Livermore National Laboratory.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Photo by Roy Kaltschmidt, courtesy of Lawrence Berkeley National Laboratory.Music by Blue Dot Sessions.
  • 176. Jessica Morse on how we can live with fire

    01:25:31
    In this Berkeley Talks episode, Jessica Morse, the deputy secretary for forest and wildland resilience at the California Natural Resources Agency, discusses the current wildfire crisis in California and how we got here, strategies the state is implementing, and lessons they've learned in order to decrease catastrophic wildfires and create more resilient forests.Morse began her Nov. 4, 2022, lecture with a story about the Camp Fire, the nation's deadliest wildfire in a century that killed 85 people and destroyed more than 18,000 structures in Northern California. "The story for me starts Nov. 8, 2018, almost four years ago to the day the campfire broke out in Paradise," began Morse. "I think all of us have some story of knowing where we were that day. It was a game-changer in terms of the deadliest, most devastating fire we've seen in California history. I went up there a couple days later to go help out and volunteer with the relief efforts. And what I saw was striking: We had 54,000 people displaced in the blink of an eye. Most of the people, even a couple days after the fire, were still meandering around in pajama bottoms and slippers because they had fled from their homes from this fire that was moving so quickly.""It was a level of trauma in people that I had not seen firsthand since I had been in an active war zone in Iraq," she continued. "And I thought, 'This is my community. These are our neighbors. This is our state. How is this happening? And how did it get this bad? And what do we do about it?'"And so, I dedicated then my time and energy, as well as many of you have to, trying to solve and answer that question. And so, I joined the Newsom administration. The governor on day one, he declared an emergency on these fires so that we could start investing in the prevention work. And so, that's what I'm going to tell you a lot about today, is how we've transformed and transitioned in California to the scale we're at today."This talk was the Berkeley Rausser College of Natural Resources' 2022 S.J. Hall Lecture in Industrial Forestry.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).UC Berkeley photo by Scott Stephens.Music by Blue Dot Sessions. 
  • 175. Siri creator Adam Cheyer shares secrets of entrepreneurship

    01:26:58
    Siri creator Adam Cheyer talks about the long road to launching the virtual assistant, how to take an entrepreneurial idea from conception to impact and why he doesn't see anything as a failure."An entrepreneur and a magician are exactly the same," begins Cheyer, who also founded the startups Change.org, Viv Labs, Sentient and Bixby. "An entrepreneur needs to imagine an impossible future. Think about Siri: 20 years ago, if I told you that you could pull a device out of your pocket, it would know who you are and where you are and you could just talk to it using your words and it would not only talk back to you, but do things for you, book that reservation, buy a movie ticket, you would've thought that were magic."An entrepreneur has to imagine an impossible future that's desirable, that doesn't exist ... So, you have to reach far as an entrepreneur, dream big, dream magical. But you have to be very clear (and answer) 'Why would we want such a thing?'"This Feb. 9, 2021, talk was part of UC Berkeley's Sutardja Center for Entrepreneurship & Technology's 2021 Newton Lecture Series. Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Music by Blue Dot Sessions.Photo courtesy of the Sutardja Center for Entrepreneurship & Technology.
  • 174. Legal scholars unpack Supreme Court ruling on affirmative action

    01:01:35
    In this episode, three leading legal scholars — john a. powell, director of UC Berkeley’s Othering & Belonging Institute (OBI); Erwin Chemerinsky, dean of Berkeley Law; and Sheryll Cashin, professor of law at Georgetown Law School — discuss the recent U.S. Supreme Court ruling that public and private universities cannot use race as a factor in admitting students. The court, with its conservative justices in the majority, ruled that such affirmative action violates the 14th Amendment to the Constitution, reversing decades of legal precedent.In California, UC Berkeley and other public colleges and universities have been prohibited from considering race in admissions since 1996, when voters approved Proposition 209.“The Supreme Court ignores the tremendous difference between using race to harm minorities as opposed to using race to remedy past discrimination and enhance diversity,” said Chemerinsky at the July 3 event, moderated by OBI Assistant Director Stephen Menendian. “When John Roberts tries to invoke Brown v. Board of Education, he ignores that Brown was dealing with laws that mandated segregation. They were all about subordinating a racial minority as opposed to what Harvard and North Carolina were doing, which was about trying to remedy past discrimination.”Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Music by Blue Dot Sessions.Photos by (from left) Howard County Library System, Brittany Hosea-Small and Sara Yogi.
  • 173. Poets laureate share works about creation, sacrifice and home

    01:18:31
    In this episode, three poets laureate — Lee Herrick, the first Asian American poet laureate of California; Kealoha, Hawai'i’s first poet laureate; and Nadia Elbgal, the Oakland youth poet laureate — perform and read their works in celebration of National Poetry Month in April.Kealoha, a slam champion who has a degree in nuclear physics from MIT, began by performing a scene from his film, The Story of Everything, a creation story inspired by his son that tells 13.8 billion years worth of time, from the Big Bang to human life on Earth. Next, Elbgal, a Yemeni American activist and recent Berkeley High graduate, read three works of hers, including "Product of a Blended Culture" and "Spark." The event concluded with a reading by Herrick, who shared several pieces from his latest book of poetry, Scar and Flower.This April 20 event was presented by UC Berkeley's Arts Research Center in partnership with Engaging the Senses Foundation, and co-sponsored by the Center for Race and Gender and the English and ethnic studies departments at Berkeley.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Music by Blue Dot Sessions.
  • 172. Biden economic adviser on building a clean energy economy

    01:20:02
    Heather Boushey, a member of President Biden’s Council of Economic Advisers (CEA) and chief economist to the Investing in America Cabinet, discusses Biden's plan to build a clean energy economy in the U.S. "The president has made clear, I feel like gazillions of times at this point, that his goal is to build an economy from the bottom up and middle out," began Boushey at the March 22 event at UC Berkeley. "He wants an economy where growth is strong, sustainable. Where gains are broadly shared. Where the economy is stable, not just strong. Where our industries are globally competitive. Where we have a strong and vibrant middle class. Where we run our economy on clean energy and we bring down carbon emissions. And where we move beyond longstanding inequities."And so, we at the CEA, we help the president as he is thinking about the economics behind how we're going to do this. And so, today's conversation, what I want to spend the next little bit of time talking to you about, is about the president's economic blueprint to reach his goals — what motivated it, what it is, why we believe that the evidence shows us that it'll be effective and what successes that we're already seeing."This talk was co-sponsored by the Berkeley Society and Economy Initiative, the Network for a New Political Economy, the Stone Center on Wealth and Income Inequality, and Social Science Matrix.Listen to the episode and read the transcript on Berkeley News (news.berkeley.edu).Executive Office of the President of the United States photo via Wikimedia Commons.Music by Blue Dot Sessions.