VR for Industrial Designers with LEAP Motion

A. Background

V2B stands for VR 2 Biz. It is a term borrowed from the B2B and B2C which are biz 2 biz and biz to consumer. Many of the VR products right now is B2C that individuals play games with it. Also, in terms of creating new objects in the VR, it is B2C products that people love to sketch and enjoy the experience of doodling. The picture below depicts the user case of the drawing in the VR.

In terms of B2B, the VR/AR is combining hand gesture to the VR/AR experience. Another user case would be an auto industry which creates an interior of the car and experience the passenger environment in the VR. The picture below shows the user using a hand to modify 3d object.

B. Accuracy

However, we need an accuracy in terms of modification.  We don’t want a dimension by eyeballing hand gestures. We need something like Autocad that has multiple digit accuracy.

In order to achieve the accuracy with hand gestures, I take a look at the micrometer. Micrometer has a dial system, a thimble and a sleeve. The thimble and sleeve  are assigned to two different scales. Sleeve is set to 0.025 inch and the thimble is set to 0.001inch.  Taking this system, I assigned each finger with different scales to achieve the accuracy.  

Here are the fingers that are assigned to the different scales.

C. 3 Axes – colors

Since the accuracy is achievable, now it needs the three dimensions. Precisely, it needs X, Y, Z. X, Y, Z are color coded with RGB at almost every software. So, XYZ::RGB is set in terms of color. So in the V2B when the axes are changed the light color changes accordingly. 

D. Hand Gestures

Since I am going to use the leap motion and the leap motion gives a distance between fingers, I take a look at the sign languages and made a chart on which distance that I need to consider. I made two versions. One is open-palm starting and the other is fist starting.

E. Feedback.

After presenting to multiple users, people gave me a feedback on the hand gestures. Most of the people understood the 3Axes because it is natural to the engineers. Engineers uses X,Y,Z with three fingers, thumb, index and middle. However, people were confused with the scale. Marek Walczak gave a nice feedback based on his architect background. Using a thumb as positive and pinky as negative. So, I finalized the hand gestures.

F. Development

During the development, I made a wooden stand that I can put the leap motion and the smartphone at the same time. It allowed both hands to work with the leap motion freely also, it allowed me to document the development with the phone camera.

Also during the demo, I found it assigning each fingers with scale creates a hurdle to modify. As a result, I simplified with plus and minus as an intro to the new interface. 

G. V2B.

Here is the final development of the V2B with the oculus and the leap motion.

Pre – AR Test

Making a Truly Synchronous Chat by Matching Word Bubbles

From the simple delivery of the letter by horse-back riding in ancient times, we know that people has demanded the synchronous information delivery between people.

However, as a social norm even if we are in the live chat with video and sound, we know that it is impossible to have the truly synchronous delivery of the information.

The maximum limit of this demand only  reach to the point where one start talking the other person has to stop in order to listen to other person.

So, here I wanted to experiment, what if we talk truly instantly by chatting with matching word bubbles.

Through out the experiment, I found that even if we have this matching system, one user waits until the other user  stops writing. It is just a matter of time human brain processing.

LINK TO THE WORK: https://dyk286.itp.io:8080/index3.html

1. BACKGROUND

We have a common sense in conversation that we keep ourselves quiet when the other person speaks. It is true that overlapping one’s voice over the other is rude. However, we have this social norm implemented in the chatting too. The chatting is synchronous platform and we can read the text as we are typing.  So I want to make synchronous text chatting.I wanted to make a new way of chatting that people keep talking and overlap their voices over the other.

2. DEVELOPMENT

I was introduced with the video and the audio which changed the thought process. Also in terms of data ownership the p2p ID generation is very clear, it is data-free(no data stored).

So, I worked on the css, html, javascript to make this project working. I used Tone.js to make sound just like other chatting webpage.  Here is the link it is working only when I run the server.js. As I am working on it I kept googling to get the answers that I want to have. As I am making this chatting, I changed it to refresh every 1minute. In this way it will make the users in the one starting point as time goes. It is the starting time of getting synchronous. So I tested the chatting with people.

3. User Test

First I wanted to have the functional completion. I chat with the friend in France and it proved the P2P is possible World-Wide and during this test I learned the lesson that in terms of video chatting, the video should go with the audio. However, in terms of drunken chat, the audio and video are constraints because it breaks the need of word bubble. So, I muted the audio and shrank the size of the video. I conceptualized it as an on-air kind sign of the broadcasting.

4. Design/Naming

I had a hard time struggling with the design and colors. At the end of the design, I made the design more seamless and made it more crisp by shadow. Also as I design the div I thought of the matching word bubbles as the fistbump. So I made a logo looks like a fist-bump.

5. Modification

I modified the design into something that I like not following Google’s material design because material design does not covey my philosophy, fun, human interactivity.

❒ LETS : Efficient Platform to Make People Social in Real World.

❒ Online services promote strengthening the physical community by joining the online community these days. However, after people joining the online community based on the physical relationship, the actual physical interaction weakens over time. We have hundreds of Facebook friends but it only ties strongly in online but not in actual life. So I wanted make an ‘efficient platform’ that quickly completes the user/group interactions with the ‘actual behavior.’

❒ In 2015, September 15 at Boston, there was a premier of the Black Mass. From this event, the one elderly woman got viral over the internet. She became an online hero because even if there was Johnny Depp she was the only person not taking picture. A futurist, Ray Kurzweil predicts digital implant in human body. He questions people’s dependency on smart phones are an example of brain extender. As we are counting on the digital devices we might lost the reality in our lives. Then, what does the digital devices provide for us? Digital devices gave us efficiency in our lives. We use the emojis to express one’s feeling as a shorthand in our text messaging. So we can see that people are looking of the efficiency in lives.

In terms of text messaging when there are multiple people, it is hard to get to the point when we are scheduling something together. One might ask for Friday night, another can’t make it on Friday, and the other is looking for other time. So, I wanted an efficient way to scheduling. Changing the process from “Talk and do co-activity” to “Decide co-activity and talk” will give a different behavior.

At ITP, people enjoy playing foosball. However finding three other players is a painful process. As time goes we made our efficient gestures. As soon as people make an eye contact both fists are shaken. As a reply, people either nodded or shaken their heads. From here, I wanted to make an efficient platform between friends. It involves a User Interface and new language. So, I wanted a simplest language to kickstart an action. This web app quickly disseminates the activities that any user wants to do to every friends. Before developing an app, I found the key how to express human activities; 5W1H. So, I trimmed it to 4, What, Where, When, Who. Each topic will have options. What: Drink, Sports, Food, Attractions. Where: 0 mile, 1 mile, 3 mile, 5mile. When: Now, 1hour later, 2hour later, 3 hour later, 4hour later. Who: 1person, 2people, 3 people, 4 people. As an user/erector, it will quickly click icons and send it to all of the friends and the friends will join the chat. In terms of language I wanted make a new application of emoji to describe a schedule.

Machine Learning will soon mock human brains in every aspects even in anything that deals with human emotions. We know many of the pop music now are computer-optimized. Industrial revolution replaced the human muscles. Now, machine learning is replacing human brains. For example, if we have nothing but autonomous driving cars, would cars need signal lights? If every car is connected, we would not even need traffic lights. In the future, something we had as a tool for communication will not be needed. So, the chatting that I am building is going for the future, you don’t talk, you don’t have to learn same language as your friend. Only thing you need is just understanding of the icons. If you watched Facebook F8 2016, the key is nothing but better expression and better share. Facebook used to have a solution to express and snapchat took over by video. Now facebook is competing against by adding a new features. So, we can see the trend how it is evolving.

Text ⇢ Picture ⇢ Video

So, I am heading to the other side of expression, even before text, the Icon. It is better sharing because they share a moment by doing same activity.

… … …

A. Background
When I was at the NYU ITP, people loved to play foosball but it has been painful to actually play… Gathering 4 people on the foosball was painful that we had go and ask people ‘you wanna play?’So, I wanted a simplest way to kickstart an action. This web app quickly disseminates the activities that an user wants to do to every friends. Before developing an app, I found the key how to express human activities; 5W1H. So, I trimmed it to 4, What, Where, When, Who. As an user/erector, it will quickly click icons and send it to all of the friends and the friends will accept/deny the message.

B. Machine Learning & Utopia
Many people questioned me about the term, ‘Utopia.’ I borrowed it from Kanye West’s interview.
“What’s your version of utopia? I don’t think people are going to talk in the future. They’re going to communicate through eye contact, body language, emojis, signs.    Imagine that. If everyone was forced to learn sign language. ”

Kanye West: Free Form

The reason why I liked the term ‘Utopia’ was because it is the future we are heading. Machine Learning will soon mock human brains in every aspects even in anything that deals with human emotions. We know many of the pop music now are computer-optimized. Industrial revolution replaced the human muscles. Now, machine learning are replacing human brains. For example, if we have nothing but autonomous driving cars, would cars need signal lights? If every car is connected, we would not even need traffic lights. In the future, something we had as a tool for communication will not be needed. So, the chatting that I am building is going for the future, you don’t talk, you don’t have to learn same language as your friend. Only thing you need is just understanding the icons.

C. Exclusive VS Inclusive
If you look at the facebook, snapchat, instagram they all have an exclusiveness. You can limit your post to your friends excluding people. It is because of facebook which started for the people who have emails ending with “.edu.” Lets that I am creating is inclusive to every friends on the list because it is action oriented not friendliness-oriented. Action oriented means instead of being behind the computer, it makes people to get together and do the activities together.

D. Expression & Share
If you watched Facebook F8 2016, the key is nothing but better expression and better share. Facebook used to have a solution to express well and snapchat took over by video. Now facebook is using instagram by adding a new feature story. So, we can see the trend how it is evolving. Text>Picture>Video. So, I am heading to the other side of expression, even before text, the Icon.

Icon ⇠   ◉   Text ⇢ Picture ⇢ Video

Instead of competing with other platforms, I am targeting the other side of expression. It is better sharing because they share a moment by doing same activity. We don’t need VR to mock being together. https://youtu.be/ouE6qyTc-l0?t=8m43s

E. Google VS Instagram
When you search do you lie what you need? When you post picture on the instagram do you post anything ugly? The data Google is collecting is pure data. People do not lie what they are searching for. On the other hand, data that instagram collects are not pure because people on the instagram want to post something that can trigger more like buttons. The data Lets will collect will be pure and sincere because you don’t lie to your friends especially when you are scheduling something together. So the future community that lets is going for is Utopian that people do not lie but are honest.

Original Design

1. Proposal
   The original plan was to make a visualization of mathematical equation. If you are familiar with Maya there is phong function. The function that changes the surface of the object in 3D. However, the Phong is named after a mathematician whose name is Phong. So, what we see on the screen is often from the mathematical equation done by some mathematicians. So, my goal was to make a visual representation of the mathematical equation.
2. Average Equation
   So, I started with a simple equation using array. The reason that I thought of using array is that because the value that influence the visual got to be changing over time. So, I started with average equation. It was easy approach because the array can be divided by the length of the array. So, mouse X values are summed up in the array and the sum of mouseX is divided by the length of the array. I divided the windowwidth into two that if the mouse X is on the left side, it will be the negative value and if the mouse X is on the right side it would be positive value. In this way the visual would be able to increase and decrease its shape over time.
3. User Testing A
   I showed this to the class and classmates were perplexed by the complexity. They did not get the what the mouseX is doing and why the visual is keep changing. I needed to explain what the logic is behind and clarify the user interaction.
4. User Testing  B
   I simplified the visual and showed it to one of the ITPers and I got a suggestion of using a webcam. So, I created the webcam and mouseX interaction. However, even after, the mouse X was not responding fast enough. I needed something that the value changes dramatically. If you put your mouse on the right side for a long time then, you would be able to see the lines moving and moving objects over the webcam will change the movements of the lines. Also if you put your mouse on the left side for a long time then, you would be able to see the lines are diminishing. https://www.doyoki.com/project/icm/finalfin3/
5. Thanksgiving Break
   I was in the bed, dreamed that I was in the darkness and saw black ellipses moving. So, I woke up and I just opened up the laptop and started to make a black ellipse changing its radius. However, the ellipse was only getting big but small. So, I changed the if statement adding “c = -c;” Now, ellipse was changing big and small.
6. Exponential Equation
   So, I played more with ellipse making more like my dream. I needed an exponential equation. So, I changed the code from  “c = c d” to “c = c d*f.” The “f” is the counter in the for loop so it would change the value in more dramatic as:
   1 1 * 1 = 2
   2 2 * 2 = 6
   6 6 * 3 = 24
   And after changing the equation, when I looked at this sphere getting big and small, I felt like I was in the space approaching to and flying away from the star.
   
7. Final
   I added more colors, more ellipses later. And I eliminated the mouse and the webcam interaction because I wanted to eliminate user’s misunderstandings. Here is the final.
   https://www.doyoki.com/project/icm/finalfin2/
   And here is the visual interpretation of code. You can see the radius of the ellipses are changing in bigger scale.
   https://www.doyoki.com/project/icm/finalfin/
   And here is the code.
   https://gist.github.com/kdoodoo/45074466d268ed84d66a
8. Conclusion
   ICM got me a better understanding of the code. I was in the computer engineering major for the first two years of college and back then I enjoyed elective courses like design 101 more rather than doing the labs on C# and  C . ICM got me better that I like coding now. And I cannot believe there are only three semesters left at the ITP and it feels much different from the first semester of the 4 years of college. I am running out of time. ITP gotta be 4 year pr0gram.

So, I have been interested in the height of the vision  that I wanted to try the vision at the lower level. So I made the VR with the vision of the dog. I used the fig rig, rope and paper cups.

First , I gave it a test shot with the fig rig. I wanted to cover the fig rig from the Theta’s view.  So I used the paper cups to cover up the fig rig and it made sense that dogs sometimes wear a collar for the medical purpose.

Second, I added a rope to make it more realistic. After the first shot, David suggested the existence of human interaction with the dog. So, he was on the shot with great acting which made the VR in great immersiveness. When I watched it with the Google cardboard, I felt like David is touching on me. I asked Paula to try it and she also said she felt like someone is touching. David brought up a perspective that there is no active will in the experience. It made the user feel passive.

Rotate the Camera View. It starts from the rear view.

The graphic is from the WEBGL of P5.JS. I made a music composed with FL Studio which is YES. The other is Grace’s recent music, You don’t own me. So, YES conveys my thought process well.

The music is continuously in loop. The musical sense is in a normal state that audience enjoys the music. Once the user is curious about the interaction, it might click. As the user click the mouse button, there is a transition to other view.

If that view is in temptation one will continuously click to explore as the the other music interferes the sense of normality.

And if one keeps clicking on the mouse the music will trans into abnormal state which will lose the sense of the music but it will be a noise.

As a result, I named the work, Anxiety Explosive.

Technical Note: I used the Tone.js, P5.js and WEBGL. The boxes are outcome of analyzer of the tone.js that the box’s XYZ dimensions are stretching and contracting and also it rotates related to the analyzer. Also, when the mouse is pressed, the camera is triggered related to mouse X and Y positions.

“People also viewed” and “People also bought” are great marketing tools of digital shopping experience. Cling to human shopping demand is a key to increase the sales. Drawing human curiosity is a real effective nudge. The other nudge we see often is “Orders over $100 are free shipping.” Making a consumer to target the amount of money is great tool. If products are mostly $99, it is much more effective.

Word processor made people to write text easily and one of the evolution we see are the webpages. Physical paper text is on the screen. List of information with human economy is truly innovative and now we love amazon and alibaba. Machine learning in these conventional platform has evolved by curating products or content like Netflix for the individuals.

How would you implement these nudges into table top artificial intelligence? At NAVER CONNECT 2018, CEO, Han Seong Sook discusses the difficulty in the artificial intelligence. “When one is shopping a milk at the supermarket, naturally the bread gets an attention but it is difficult in artificial intelligence. Ordering a milk through the table top speaker is just a simple declaration.”

Consumers are rational when they face Alexa. They give orders and that is it. However, if we tweak “Consumers are rational” we also know consumers are irrational. This is a simple finding we learned over the decades. We can see this irrationality through game. People buy items for their personified avatars in the game. In japan we see otakus getting married with dolls. Personification of Alexa and the ability to individualize Alexa can draw more value to the consumers. Just like consumers spend money on their virtual characters in the game.

The other option is present in chatting application. The facebook messenger is suggesting food delivery while two users are talking about having lunch. The only downfall is they were discussing having a lunch at the restaurant. Just like allowing access location of the smartphone, we might need to allow an access to our everyday conversation. Instead of calling out the names, “Alexa” or “OK, google” we might have to let the artificial intelligence listen our everyday conversation. Soon, Alexa will intervene our conversation: “Any delivery for the lunch?”

Gamification in marketing is effective in games. However, monetization by the individualizing an avatar made an instant growth in sales but failed over time just like cyworld by SK in Korea. As soon as the user feels lame to decorate its virtual avatar, soon it will lose the users. Keeping clean and less messy with advertising is a key in user experience but we cannot keep it too dry which cannot draw any attention to bread when it is buying a milk. Letting artificial intelligence to listening to our everyday conversation is controversial. We know keeping a mouth shut is an important role of a butler from the Downton Abbey.

We might face the need of an auto-mode. Just like monthly curation and subscription, we can auto our lives. Auto weekly food delivery and auto monthly clothes delivery based on artificial intelligence analysis. Soon Alexa will deliver your bread right next to your milk. 

FULL SCREEN1

FULL SCREEN2

Samsung recently launched Bixby where it resides in the smartphone just like its popularity in smartphones. Where does Samsung make sales? How would Bixby create sales?  Samsung already has a successful platform Samsung Pay. Samsung should start implement Samsung Pay into Bixby. All other white appliance should also have both Bixby and Samsung Pay. If it is a refrigerator with Bixby and Samsung Pay, it can instantly work with other consumer industries to order products. Galaxy phones already have a biometric data collecting like iris, finger print scanner which Alexa cannot compete against.

Autonomous driving is another form of artificial intelligence and people in the industry already prescribed HaaS(Hardware as a Service). GM is moving forward with subscription service in Cadillac brand. Consumers can subscribe cars and change cars monthly. The benefit of autonomous driving is well structured in this model because it can create direct sales. Subscribers can order cars and the autonomous cars will be delivered, basically will deliver by itself. Making sales out of the artificial intelligence is a key to success and companies like Amazon, GM which delivers the tangible products are moving forward looking at where to make sales.

Human Muscle interpretation in robotic arm mechanism.

Motorized device has a limited mimicry on human/animal muscle movement. Being precise to satisfy industrial demands has allowed robots to replace human labor.

However, it lost a sense of human mimicry in natural movement. Just like musicians avoid exact tempo of the instruments at the MIDI software to make more humane sound,  the movement of the robot hand mocks the human natural motion.

Implementing humanness on the robotic hand is based on the long time ago animal experiment. From this well known experiment on the dead body of the frog,  we can see animal muscle is triggered by electrons.

Making frogs’ leg muscle popping by connecting electricity clearly shows disgust of the combination of the nature and the artificial implant. However, from the scientific sense of  the experiment, it allowed me to make this human-like robot arm.

Instead of digital input to analog output, here both are analog reading and analog output. Magnet was used to convey natural human instability in human motion. Instability means imperfection which is a contrary to industrial robots with G-code.

As one can see, the servo motor is shaking as the magnetic flux sensor is approaching to the magnet.  Just like when human limbs are responding to the objects ahead of touching them directly.

The exploration of this human mimicry will lower the barrier of the denial against artificial robots.