The thrilling potential of SixthSense technology | Pranav Mistry

We grew up interacting with the physical
objects around us. There are an enormous number of them
that we use every day. Unlike most of our computing devices, these objects are much more fun to use. When you talk about objects, one other thing automatically
comes attached to that thing, and that is gestures: how we manipulate these objects, how we use these objects in everyday life. We use gestures not only to interact
with these objects, but we also use them
to interact with each other. A gesture of “Namaste!”,
maybe, to respect someone, or maybe, in India I don’t need to teach
a kid that this means “four runs” in cricket. It comes as a part
of our everyday learning. So, I am very interested,
from the beginning, how our knowledge
about everyday objects and gestures, and how we use these objects, can be leveraged to our interactions
with the digital world. Rather than using a keyboard and mouse, why can I not use my computer in the same way that I interact
in the physical world? So, I started this exploration
around eight years back, and it literally started
with a mouse on my desk. Rather than using it for my computer,
I actually opened it. Most of you might be aware
that, in those days, the mouse used to come with a ball inside, and there were two rollers that actually guide the computer
where the ball is moving, and, accordingly,
where the mouse is moving. So, I was interested in these two rollers, and I actually wanted more, so I borrowed
another mouse from a friend — never returned to him — and I now had four rollers. Interestingly, what I did
with these rollers is, basically, I took them off of these mouses
and then put them in one line. It had some strings
and pulleys and some springs. What I got is basically
a gesture-interface device that actually acts
as a motion-sensing device made for two dollars. So, here, whatever movement
I do in my physical world is actually replicated
inside the digital world just using this small device
that I made, around eight years back, in 2000. Because I was interested
in integrating these two worlds, I thought of sticky notes. I thought, “Why can I not connect the normal interface
of a physical sticky note to the digital world?” A message written
on a sticky note to my mom, on paper, can come to an SMS, or maybe a meeting reminder automatically syncs
with my digital calendar — a to-do list that automatically
syncs with you. But you can also search
in the digital world, or maybe you can write a query, saying, “What is Dr. Smith’s address?” and this small system
actually prints it out — so it actually acts like a paper
input-output system, just made out of paper. In another exploration, I thought of making a pen
that can draw in three dimensions. So, I implemented this pen
that can help designers and architects not only think in three dimensions, but they can actually draw, so that it’s more intuitive
to use that way. Then I thought,
“Why not make a Google Map, but in the physical world?” Rather than typing a keyword
to find something, I put my objects on top of it. If I put a boarding pass, it will show me
where the flight gate is. A coffee cup will show
where you can find more coffee, or where you can trash the cup. So, these were some
of the earlier explorations I did because the goal was to connect
these two worlds seamlessly. Among all these experiments, there was one thing in common: I was trying to bring
a part of the physical world to the digital world. I was taking some part of the objects, or any of the intuitiveness of real life, and bringing them to the digital world, because the goal was to make
our computing interfaces more intuitive. But then I realized that we humans
are not actually interested in computing. What we are interested in is information. We want to know about things. We want to know about
dynamic things going around. So I thought, around last year —
in the beginning of the last year — I started thinking, “Why can I not take
this approach in the reverse way?” Maybe, “How about I take my digital world and paint the physical world
with that digital information?” Because pixels are actually, right now,
confined in these rectangular devices that fit in our pockets. Why can I not remove this confine and take that to my everyday
objects, everyday life so that I don’t need
to learn the new language for interacting with those pixels? So, in order to realize this dream, I actually thought of putting
a big-size projector on my head. I think that’s why this is called
a head-mounted projector, isn’t it? I took it very literally, and took my bike helmet, put a little cut over there so that
the projector actually fits nicely. So now, what I can do — I can augment the world around me
with this digital information. But later, I realized that I actually
wanted to interact with those digital pixels, also. So I put a small camera over there
that acts as a digital eye. Later, we moved to a much better, consumer-oriented pendant version of that, that many of you now know
as the SixthSense device. But the most interesting thing
about this particular technology is that you can carry
your digital world with you wherever you go. You can start using any surface,
any wall around you, as an interface. The camera is actually tracking
all your gestures. Whatever you’re doing with your hands, it’s understanding that gesture. And, actually, if you see,
there are some color markers that in the beginning version
we are using with it. You can start painting on any wall. You stop by a wall,
and start painting on that wall. But we are not only tracking
one finger, here. We are giving you the freedom
of using all of both of your hands, so you can actually use both of your hands
to zoom into or zoom out of a map just by pinching all present. The camera is actually doing —
just, getting all the images — is doing the edge recognition
and also the color recognition and so many other small algorithms
are going on inside. So, technically,
it’s a little bit complex, but it gives you an output which is more
intuitive to use, in some sense. But I’m more excited that you can
actually take it outside. Rather than getting your camera
out of your pocket, you can just do the gesture
of taking a photo, and it takes a photo for you. (Applause) Thank you. And later I can find a wall, anywhere, and start browsing those photos or maybe, “OK, I want to modify
this photo a little bit and send it as an email to a friend.” So, we are looking for an era where computing will actually merge
with the physical world. And, of course,
if you don’t have any surface, you can start using your palm
for simple operations. Here, I’m dialing a phone number
just using my hand. The camera is actually not
only understanding your hand movements, but, interestingly, is also able to understand what objects
you are holding in your hand. For example, in this case, the book cover is matched with so many thousands,
or maybe millions of books online, and checking out which book it is. Once it has that information, it finds out more reviews about that, or maybe New York Times
has a sound overview on that, so you can actually hear,
on a physical book, a review as sound. (Video) Famous talk
at Harvard University — This was Obama’s visit last week to MIT. (Video) And particularly I want
to thank two outstanding MIT — Pranav Mistry: So, I was seeing
the live of his talk, outside, on just a newspaper. Your newspaper will show you
live weather information rather than having it updated. You have to check your computer
in order to do that, right? (Applause) When I’m going back,
I can just use my boarding pass to check how much my flight
has been delayed, because at that particular time, I’m not feeling like opening my iPhone, and checking out a particular icon. And I think this technology
will not only change the way — (Laughter) Yes. It will change the way
we interact with people, also, not only the physical world. The fun part is, I’m going
to the Boston metro, and playing a pong game inside the train
on the ground, right? (Laughter) And I think the imagination
is the only limit of what you can think of when this kind of technology
merges with real life. But many of you argue, actually, that all of our work is not
only about physical objects. We actually do lots
of accounting and paper editing and all those kinds of things;
what about that? And many of you are excited
about the next-generation tablet computers to come out in the market. So, rather than waiting for that, I actually made my own,
just using a piece of paper. So, what I did here
is remove the camera — All the webcam cameras have
a microphone inside the camera. I removed the microphone from that, and then just pinched that — like I just made a clip
out of the microphone — and clipped that to a piece of paper,
any paper that you found around. So now the sound of the touch is getting me when exactly
I’m touching the paper. But the camera is actually tracking
where my fingers are moving. You can of course watch movies. (Video) Good afternoon.
My name is Russell, and I am a Wilderness
Explorer in Tribe 54.” PM: And you can of course play games. (Car engine) Here, the camera is actually understanding
how you’re holding the paper and playing a car-racing game. (Applause) Many of you already must have
thought, OK, you can browse. Yeah. Of course you can
browse to any websites or you can do all sorts
of computing on a piece of paper wherever you need it. So, more interestingly, I’m interested in how we can
take that in a more dynamic way. When I come back to my desk,
I can just pinch that information back to my desktop so I can use my full-size computer. (Applause) And why only computers?
We can just play with papers. Paper world is interesting to play with. Here, I’m taking a part of a document, and putting over here a second part
from a second place, and I’m actually modifying the information
that I have over there. Yeah. And I say, “OK, this looks nice,
let me print it out, that thing.” So I now have a print-out of that thing. So the workflow is more intuitive, the way we used to do it
maybe 20 years back, rather than now switching
between these two worlds. So, as a last thought, I think that integrating
information to everyday objects will not only help us to get rid
of the digital divide, the gap between these two worlds, but will also help us, in some way, to stay human, to be more connected
to our physical world. And it will actually help us
not end up being machines sitting in front of other machines. That’s all. Thank you. (Applause) Thank you. (Applause) Chris Anderson: So, Pranav,
first of all, you’re a genius. This is incredible, really. What are you doing with this?
Is there a company being planned? Or is this research forever, or what? Pranav Mistry: So, there are
lots of companies, sponsor companies of Media Lab
interested in taking this ahead in one or another way. Companies like mobile-phone operators want to take this in a different way
than the NGOs in India, thinking, “Why can we only
have ‘Sixth Sense’? We should have a ‘Fifth Sense’
for missing-sense people who cannot speak. This technology can be used for them
to speak out in a different way maybe a speaker system.” CA: What are your own plans?
Are you staying at MIT, or are you going to do
something with this? PM: I’m trying to make this
more available to people so that anyone can develop
their own SixthSense device, because the hardware is actually
not that hard to manufacture or hard to make your own. We will provide all the open source
software for them, maybe starting next month. CA: Open source? Wow. (Applause) CA: Are you going to come back to India
with some of this, at some point? PM: Yeah. Yes, yes, of course. CA: What are your plans? MIT? India? How are you going to split
your time going forward? PM: There is a lot of energy here.
Lots of learning. All of this work that you have seen
is all about my learning in India. And now, if you see, it’s more about
the cost-effectiveness: this system costs you $300 compared to the $20,000 surface tables,
or anything like that. Or maybe even the $2 mouse gesture system
at that time was costing around $5,000? I showed that, at a conference,
to President Abdul Kalam, at that time, and then he said, “OK, we should use this
in Bhabha Atomic Research Centre for some use of that.” So I’m excited about how I can bring
the technology to the masses rather than just keeping that technology
in the lab environment. (Applause) CA: Based on the people we’ve seen at TED, I would say you’re truly
one of the two or three best inventors in the world right now. It’s an honor to have you at TED. Thank you so much. That’s fantastic. (Applause)

You may also like...

86 Responses

  1. Vishwa says:

    This video will change the way you look at your life.

  2. R D says:

    philosophy for dummies


    Awesome dude! I am speechless. Hats off to you.

  4. MickFinn says:

    Interesting how people need to comment and highlight him being an Indian. I don't see other comments saying proud of my Caucasian brother or the like….

  5. Kells Kelly says:

    Does he know coding?

  6. harshluhar says:

    So this was 10 years ago?! ๐Ÿ˜ฑ

  7. Stella of the Lake says:

    "We human are not interested in computing but we're interested in information."


  8. Shruti Bhatt says:

    Touch screen technology without and ahead smart phone ๐Ÿ‘๐Ÿ‘๐Ÿ‘๐Ÿ‘

  9. Jacob Opstad says:

    What an elegant idea! Though that projector can really only be used in dark places.

  10. Josh R says:

    That table looks like a very early version of Microsoft Surface

  11. Benjamin Dixon says:

    I have a better idea than finger condoms, get in touch

  12. avni 07 says:

    Next Elon Musk…๐Ÿ‘๐Ÿ‘๐Ÿ‘

  13. Eshaan Rathi says:

    man, Pranav u have got excellent vision and creative ideas

  14. Gamers War game says:

    Indians are genius.

  15. shinu siri says:


  16. Jason Todd says:

    It's 2019

  17. Trevor Mwendwa says:


  18. Aditya Pagare says:

    And this was 10 years ago….

  19. Dhananjay Mishra says:

    You are my favourite

  20. Dhananjay Mishra says:

    This is amazing How someone can someone use a mouse part to create such an amazing invention

  21. Dhananjay Mishra says:

    More than a Legend

  22. Xe M says:

    Is this the guy who inspired Microsoft HoloLens and Google Glasses?

  23. errasch says:

    Successful Community Advocates. Pick Yourself Up has guest Eric Rasch, Operations Manager
    at the Silver Spring Civic Building at Veterans Plaza.ย Video:

  24. Masi jan says:

    Wowwwwwoooow this great u r the human love you bro God bless you


    He is totally outstanding and amazing.He converted the physical information into digital one by just contracting the fingers.HATS OFF TO SUCH GENIUS PEOPLE.Proud to an Indian

  26. harish babu says:

    Great innovator…worth watching

  27. Justina says:

    Check this video of his progress as of 2017. It is actually hilarious and maybe a little sad.

  28. Biscope เฆฌเฆพเฆฏเฆผเง‹เฆธเงเฆ•เฆช says:

    That was incredible!
    It's 2019 now! I'm wondering what he's upto nowadays

  29. Murlidhar shridhar says:

    Incredible at another different level.proud on you sir u r working for masses..

  30. Srinivasa Anirudh Bellamkonda says:

    The current mobile phone technology will reach a saturation point soon from a design standpoint, consumers will soon become impatient and will want something to satisfy their subconscious hunger and addiction for new technology. Samsung will then strategically introduce the basic version of this technology concept to attain a leading position in the global market. This tech was introduced more than 10 years ago. Think about it.

  31. Adi Sharma says:


  32. M Jayanthi says:

    Wow amazing

  33. Satej Hake says:

    Anyone in 2020 ?

  34. Yamini V says:

    Full form of ted

  35. Krishna Prabhu says:

    Wow great

  36. Samuel Wu says:

    Still waiting for it.

  37. R K BHINDER says:

    my ideal

  38. Michael Fenger Brandt says:

    … genius …

  39. Ember2460 Harrington says:

    Honking car locking wakes people up and causes noise related stress. It can be heard from behind the walls of the buildings you park next to day and night. Please lock and arm you cars quietly with light flash only for your neighbors sleep and peace of mind. Thank you. ๐Ÿ™

  40. CrackingCodingInterview says:

    is it in infosys mysore campus?

  41. Atharva Kangude says:

    The only question is Why these things are not in mainstream and why is not famous why why why

  42. SANGITA SHETE says:

    Samsung Bixby might be inspired from this.. as he is the head of Research at Samsung

  43. Culture Bearer says:


  44. Parth says:

    I googled about him ..and i found..that he is belongs to my city…thats amazing..even i font know him

  45. vandana rao says:


  46. Al Alaya Jasmine says:

    Exceptionally enchanting

  47. รœbermensch says:

    Problem with this world, it has only 33k views

  48. offensiven't malayalam memes says:

    There are more than six senses

  49. Armaan Verma says:

    This guy has made history and will keep doing that why is he so underrated he has done something better than some big scientists

  50. Terri Brown says:

    This was 10 yrs ago… Where is all this tec? I want it all!!! ๐Ÿ˜œ

  51. Paul Taylor says:

    I wish I would have went into technology as a career. Medical field blows.

  52. Mohanlal Sangappillai says:

    hololens tech starts from him?

  53. Cyrus Payne says:

    Published in 2009. YouTube suggests it in 2019. Way to go software development experts! xD

  54. Sai Krishna says:

    This is 9 years old. What happened? Where is this tech? What are you waiting for?

    Any update?

  55. Vasu Chauhan says:

    He's graduated from Nirma university, Ahmedabad.

  56. marvel officials says:

    proud to be indian

  57. Magdy Salah AboSalem says:

    Wonderful genius

  58. lewis works says:

    This amazing guy is ruined by Samsung, what a pity!!!!

  59. Nghia Hung says:

    It s been 10 years and and we re still far from this future.

  60. Dejanae Oakley- Joseph says:

    This is fucking revolutionary

  61. Bit Tube says:

    Hey TED,
    joyce31004 has sent 1 Tube to you. In order to receive your donation check out bittubeapp(dot)com

  62. Pooja Mourya says:

    Now lam big fan of u pranav sir ๐Ÿ™‚
    I used watch this amazing things in movies and think, I wish I could do Thea's things in real.but after watching this wo..w .not iam but at least some one made it real l
    am proud to watch this๐Ÿ˜๐Ÿ˜๐Ÿ˜๐Ÿ˜

  63. Priti Mohanmala says:

    It was really amazing for me and thank you pranav sir for the regarding this

  64. Mintu Patel says:

    Another Albert Einstein

  65. Samriddhi Agarwal says:

    Proud to be an Indian . His thinking of connecting digital world to physical world moved me today.
    Thank You ๐Ÿ‘

  66. Vanita Sagar says:

    Pranav is from nirma university
    I am also from nirma university

  67. taneesh Bhojawala says:

    It was a very inovative way to bring the digital world into the physical world.

  68. taneesh Bhojawala says:

    Such a nice video. Its very inspiring and it made me think to make something inovative and useful like this.

  69. Jacob David Cunningham says:

    10:25 there's like now way that would work, you'd need super high quality camera to read the characters unless you had a digital copy of the book/knew what page you were on and what section to grab. OCR/OpenCV/ML/Human people manually reading/typing info
    the real time copy/paste could be a simple image of what it saw

  70. Vishnu Khattri says:


  71. Sangeet says:

    I can't believe he thought of this 10 years ago. Incredible.

  72. S.N. Bose says:

    His technology not come up yet because he will end up all the mobile company .

  73. Jimm McCord says:

    His carelessness to speak & learn our language causes a distraction. You must learn to enunciate English to grow your base.

  74. Tharun gupta says:


  75. Tharun gupta says:

    New Avenger , after death of Tony stark
    Hoo cap here's a new iron man ๐Ÿ˜‚

  76. Dimon4k Top says:

    If you want to earn every day 20 euro go there!!!

  77. Ajesh DS says:

    At 2009 this guy was living in 3009

  78. Stan Mann says:

    Fake science from fake scientists. Why don't you tell us what your rock stars said about the gay gene Ted?

  79. INDIAN LIVE says:

    The Real Tony starck

  80. omni light_Xl says:

    I saw this video as a 12 year old kid

  81. Saurabh Shrivastava says:

    Why this was never released to market ??

  82. sriram sakthivel says:


  83. Jim Green says:

    AND….while people play with these 'toys' at times they will see and learn IDEAS of concepts that are….possible [?]. AND as they keep playing, and the concepts continue to show themselves. Eventually they will catch fire in the mind of those few….those many…….and the SIXTH SENSE….the FOURTH DIMENSION will be all that much closer! Ah, but yet I ramble. Ah, but yet…I believe!

  84. Bhanu Vaishnav says:

    Proud to be i am palanpuri…

  85. Franklin B says:

    Perfect applications for AR

  86. Jeanita Blue says:

    This is amazing! I want to get my hands on these products! #2019Watching

Leave a Reply

Your email address will not be published. Required fields are marked *