3D Scanning  

Microsoft is developing new real-time 3D scanning capabilities that could mean you could attend a concert or sporting event live in full 3D, or even have the ability to communicate in real-time with remotely captured people using immersive augmented reality or virtual reality displays in the future.

Researchers at Microsoft have created a system that could be the prototype for a next-generation Kinect camera. Called Fusion4D, the project, the scanning system impressively reconstructs complex 3D scenes digitally, including those with more than one person, animals and can even capture clothing being put on the actor.

The researchers have detailed their work in a paper published online.

Fusion4D is the first real-time multi-view non-rigid reconstruction system for live performance capture, claim the researchers. "We have contributed a new pipeline for live multi-view performance capture, generating high-quality reconstructions in real-time, with several unique capabilities over prior work," they conclude.

Related articles

• 3D Scan the World for Robots 
• Turn Your iPad into a 3D Scanner 
• Microsoft Achieves Substantial Beyond-Human-Level Deep Learning Advance 

Today, most cameras and 3D scanners like the Kinect Sensor, used for motion capture still focus on static, non-moving, scenes. This is due to limitations in computational power and the demands on software to reconstruct scenes. 

For more complex scenes, with moving cameras and many elements, the computer must solve for orders of magnitude more parameters in real-time. This typically results in noisy or missing data, choppy motion and digital artifacts in the output that are not representative of what is being captured in the real world.

"Our reconstruction algorithm enables both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion."

Microsoft's research team also dealt with the case of changing scene topology, such as person removing a jacket or scarf.

The implications of the research are vast. For instance, it could lead to new real-time experiences such as the ability to watch a remote concert or sporting event live in full 3D, or even the ability to communicate in real-time with remotely captured people using immersive augmented reality or virtual reality displays.

The applications could also extend to robotics and machine vision.

With Microsoft's HoloLens system reaching wider deployment now, this last case could lead to some very interesting possibilities.

"As shown, our reconstruction algorithm enables both incremental reconstruction, improving the surface estimation over time, as well as parameterizing the nonrigid scene motion," write the authors."We also demonstrated how our approach robustly handles both large frame-to-frame motion and topology changes. This was achieved using a novel real-time solver, correspondence algorithm, and fusion method."

"We believe our work can enable new types of live performance capture experiences, such as broadcasting live events including sports and concerts in 3D, and also the ability to capture humans live and have them re-rendered in other geographic locations to enable high fidelity immersive telepresence."

SOURCE  Microsoft Research

By 33rd Square

Embed

 Androids  

A university student in Japan has a created lifelike humanoid robot of Yuki Kashiwagi, a singer in Japanese girl idol group.

A Japanese university student has made a lifelike humanoid robot of Yuki Kashiwagi, a singer from the massively popular Japanese girl group AKB48. The idol group one of the highest-earning musical acts in Japan and is considered to be a modern social phenomenon.

Displayed at the 2014 Nico Nico ChoKaigi Super Conference in Tokyo, the "Yukirin Robot" depicts Kashiwagi in a long white dress, and an Xbox Kinect sensor built into the chest of the robot reacts to humans nearby, so that the robot turns and fixes her gaze on anyone that enters her field of vision.

Related articles Robot Revolution - Will Machines Surpass Humans?  KIROBO Robot Prepares For Work Aboard The International Space Station MIT Researcher Teaches Robots To Adapt

According to Japanese website IT Media Japan, student Takayuki Todo says that he carved the robot's face out of soft wood by looking at magazine photographs.

The real Yuki Kashiwagi

Todo plans to upgrade the robot with better head tracking, eye pupils and more expressive eyelids.

While obviously skirting the uncanny valley (especially with no arms), this robot is an impressive achievement for a student working alone.




SOURCE  International Business Times

By 33rd Square

Embed

 Augmented Reality  

Augmented reality company Metaio is showing off a new feature that will be useful for trying on glasses and sunglasses virtually.  As AR technology continues to develop tools like this will get better and be included on mobile devices too.

M any people don’t realize that the all-time fastest selling consumer electronic device is at the core level, a 3D depth camera. The Microsoft Kinect was the first major release of a device that had the power to understand its surroundings, detect movement and gestures and even identify real world objects – and it was really just a companion to an already popular gaming device.

Related articles Augmenting Human Capability  Not Quite Full Virtual Reality, but Google Getting Close  New Kinect Specs Leaked

The overwhelming success of the Kinect Sensor began a steady rise in interest in what 3D and depth cameras could do for consumer electronics.

Now, other companies like Apple, Intel and NVIDIA shortly announced what most of the mobile industry had been anticipating-that they too, had been working on 3D depth cameras. Apple recently acquired the company behind Kinect, PrimeSense.  Their focus wasn’t gaming, but rather taking the same technology and making it mobile- smaller, and embedded into future mobile devices.

“3D cameras will soon arrive on mobile devices,” said augmented reality company Metaio CTO Peter Meier recently. “Developers and businesses alike will be able to take advantage of this new technology through the support for 3D cameras in the Metaio SDK. This year in Barcelona we are showcasing how we have adapted our core technology to support this new wave of 3D integration, while also demonstrating the power of silicon integration, especially with regard to a wearable future. AR has already shown usefulness and value in both enterprise and consumer sectors, and we will continue to lead the way in innovation for 2014.”

In the video demonstration below, recorded at the Metaio booth at Mobile World Congress 2014, a real-time 3D reconstruction of a face is used to try out different models of sunglasses.  The system uses a Microsoft Kinect camera to generate the model.  While this version is for desktop, it will soon be available for mobile devices as the technology expands and progresses.



SOURCE  Metaio AR

By 33rd Square

Subscribe to 33rd Square

 Business  

Apple has acquired the company PrimeSense, the Israel-based company behind the gesture sensing technology of the original Xbox Kinect. At this time, Apple has not revealed their plans for the company or how the technology might be incorporated into the next iPhone or Apple TV.

T he company that helped develop the technology behind the original Microsoft Xbox Kinect device is now a part of Apple.  Israel-based PrimeSense Ltd, makers of sensors that enable three-dimensional machine vision was reportedly bought out for about $350 million.

An Apple spokesman said, "Apple buys smaller technology companies from time to time, and we generally do not discuss our purpose or plans."

A spokeswoman for PrimeSense said: "We can confirm the deal with Apple. Further than that, we cannot comment at this stage."

PrimeSense's sensing technology, which gives digital devices the ability to observe a scene in three dimensions, was used to help power Microsoft's Xbox Kinect.

PrimeSense’s motion sensor technology now powers over 24 million devices around the world.

Related articles Apple Patent Hints At Iron Man Style CAD Interface  Use Gesture Interfaces Throughout Your Home Using Only a WiFi Signal  Microsoft Kinect Research Points To Future Natural Interface Applications

The technology has increased the use of gesture or "natural interfaces," and has encouraged a new breed of hackers.  Kinect sensors have also been found to be very useful in robotics applications, including SLAM (Simultaneous Location and Mapping) functions.

Apple's interest in PrimeSense was first reported in July by Israeli financial newspaper Calcalist.

PrimeSense has expanded its product line to include more hardware than the original large, stationary sensor seen in the Kinect in the past few years. These developments incluce creating new, smaller sensors targeted at more compact devices. The company’s Capri model, for example, seems particularly well suited for the mobile market, potentially now being incorporated into the next generation iPhone.



SOURCE  All Things D

By 33rd Square

Subscribe to 33rd Square

 Virtual Reality  

Researcher Oliver Kreylos has been working with Virtual Reality for over a decade.  In his recent work, he has combined Oculs Rift headset, Razer Hydra joysticks and Kinect camera to create a virtual environment with a lot of promise.

Oliver Kreylos, a researcher at UC Davis has been working on immersive virtual environments for more than a decade. What you’ll see in the video above is a combination of three different technologies, all originally intended for gaming, but here set up to explore new interface paradigms: A set of Oculus Rift goggles to project a 3D environment directly into Kreylos’s eyes and track the orientation of his head, two Razer Hydra joysticks to track the movement of his hands and give him some buttons to press, and a Kinect motion tracking device.

Related articles Use Gesture Interfaces Throughout Your Home Using Only a WiFi Signal  Developer Adds Arm Control To Oculus Rift Virtual Reality System  Microsoft Kinect Research Points To Future Natural Interface Applications

The use of the Kinect is especially interesting.  As Kreylos found, using the Oculus Rift headgear blocked the vision of his hands and his other devices, like the keyboard and mouse.  Overlaying 3D video from the Kinect into his VR environment, allows him to see his hands in space properly as well as other items on his desktop.

This video really demonstrates how augmented reality and virtual reality are merging and may dramatically impact how certain computer applications (as well as gaming) could evolve in the future.  As anyone working with 3D CAD systems will be aware, there is always a disconnect between the on-screen version of a 3D file and the real-world equivalent.  (This is, of course, a boon for the 3D printing industry...)

With research like that done by Kreylos, the design world may be about to make another leap, as dramatic as the one between 2D drawing and the advent of 3D CAD systems.

To demonstrate this, Kreylos also produced another video (above), which he states provides an effective benchmark for the effectiveness of the new Oculus Rift-enabled VR system.  In it, he creates a 3D dimensional Buckyball from individual digital carbon atoms. According to Kreylos this task can take as long as 45 minutes with a keyboard and a mouse, but a good 3D environment allows a person to do it in under two minutes.

The advent of a consumer-grade, affordable virtual reality system has led softare pioneer John Carmack to join Oculus as CTO.

For more about Kreylos’ work, it’s worth checking out his ongoing series of blog posts on the Oculus Rift and the challenges of virtual reality in general.

SOURCE  Quartz

By 33rd Square

Subscribe to 33rd Square

 Computer Vision  

Researchers at Carnegie Mellon University have developed a method for tracking the locations of multiple individuals in complex, indoor settings using a network of video cameras, creating something similar to the fictional Marauder’s Map used by Harry Potter to track comings and goings at the Hogwarts School.

Researchers at Carnegie Mellon University have developed a method for tracking the locations of multiple individuals in complex, indoor settings using a network of video cameras, creating something similar to the fictional Marauder's Map used by Harry Potter to track comings and goings at the Hogwarts School.

The method used in the research was able to automatically follow the movements of 13 people within a nursing home, even though individuals sometimes slipped out of view of the cameras. None of Potter's magic was needed to track them for prolonged periods; rather, the researchers made use of multiple cues from the video feed: apparel color, person detection, trajectory and, perhaps most significantly, facial recognition.

Related articles Google and NASA Team Up To Explore the Potential of the D-Wave Quantum Computer  AI System Uses Internet Image Searches To Identify Objects  Use Gesture Interfaces Throughout Your Home Using Only a WiFi Signal

Multi-camera, multi-object tracking has been an active field of research for a decade, but automated techniques have only focused on well-controlled lab environments. The Carnegie Mellon team, by contrast, proved their technique with actual residents and employees in a nursing facility—with camera views compromised by long hallways, doorways, people mingling in the hallways, variations in lighting and too few cameras to provide comprehensive, overlapping views.

The performance of the Carnegie Mellon algorithm significantly improved on two of the leading algorithms in multi-camera, multi-object tracking. It located individuals within one meter of their actual position 88 percent of the time, compared with 35 percent and 56 percent for the other algorithms.

The researchers—Alexander Hauptmann, principal systems scientist in the Computer Science Department (CSD); Shoou-I Yu, a Ph.D. student in the Language Technologies Institute; and Yi Yang, a CSD post-doctoral researcher—will present their findings June 27 at the Computer Vision and Pattern RecognitionConference in Portland, Ore.

Though Harry Potter could activate the Marauder's Map only by first solemnly swearing "I am up to no good," the Carnegie Mellon researchers developed their tracking technique as part of an effort to monitor the health of nursing home residents.

"The goal is not to be Big Brother, but to alert the caregivers of subtle changes in activity levels or behaviors that indicate a change of health status," Hauptmann said. All of the people in this study consented to being tracked.

These automated tracking techniques also would be useful in airports, public facilities and other areas where security is a concern. Despite the importance of cameras in identifying perpetrators following this spring's Boston Marathon bombing and the 2005 London bombings, much of the video analysis necessary for tracking people continues to be done manually, Hauptmann noted.

The CMU work on monitoring nursing home residents began in 2005 as part of a National Institutes of Health-sponsored project called CareMedia, which is now associated with the Quality of Life Technology Center, a National Science Foundation engineering research center at CMU and the University of Pittsburgh.

"We thought it would be easy," Hauptmann said of multi-camera tracking, "but it turned out to be incredibly challenging."

Something as simple as tracking based on color of clothing proved difficult, for instance, because the same color apparel can appear different to cameras in different locations, depending on variations in lighting. Likewise, a camera's view of an individual can often be blocked by other people passing in hallways, by furniture and when an individual enters a room or other area not covered by cameras, so individuals must be regularly re-identified by the system.

Face detection helps immensely in re-identifying individuals on different cameras. But Yang noted that faces can be recognized in less than 10 percent of the video frames. So the researchers developed mathematical models that enabled them to combine information, such as appearance, facial recognition and motion trajectories.

Using all of the information is key to the tracking process, but Yu said facial recognition proved to be the greatest help. When the researchers removed facial recognition information from the mix, their on-track performance in the nursing home data dropped from 88 percent to 58 percent, not much better than one of the existing tracking algorithms.

The nursing home video analyzed by the researchers was recorded in 2005 using 15 cameras; the recordings are just more than six minutes long.

Further work will be necessary to extend the technique during longer periods of time and enable real-time monitoring. The researchers also are looking at additional ways to use video to monitor resident activity while preserving privacy, such as by only recording the outlines of people together with distance information from depth cameras similar to the Microsoft Kinect.

SOURCE  Carnegie Mellon University

By 33rd Square

Subscribe to 33rd Square

 Gesture Control  

University of Washington researchers have shown it's possible to leverage wi-fi signals around us to detect specific movements without needing sensors on the human body or cameras. Using a wi-fi router and a few wireless devices in the living room, users could control their electronic devices from any room in the home with a simple gesture.

Forget to turn off the lights before leaving the apartment? No problem. Just raise your hand, finger-swipe the air, and your lights will power down. Want to change the song playing on your music system in the other room? Move your hand to the right and flip through the songs.  Sound like a fantasy?  It may not be according to researchers at the University of Washington.

University of Washington computer scientists have developed gesture-recognition technology that brings this a step closer to reality. Researchers have shown it’s possible to leverage Wi-Fi signals around us to detect specific movements without needing sensors on the human body or cameras.

By using an adapted Wi-Fi router and a few wireless devices in the living room, users could control their electronics and household appliances from any room in the home with a simple gesture.

Related articles Sci-fi Destroys the Future, Science Builds It According To Daniel H. Wilson  Robots Reach Through a Cluttered World  Researchers Fly Robot Drone With Their Thoughts

“This is repurposing wireless signals that already exist in new ways,” said lead researcher Shyam Gollakota, a UW assistant professor of computer science and engineering. “You can actually use wireless for gesture recognition without needing to deploy more sensors.”

The UW research team that includes Shwetak Patel, an assistant professor of computer science and engineering and of electrical engineering and his lab, published their findings online this week. This technology, which they call “WiSee,” is to appear at The 19th Annual International Conference on Mobile Computing and Networking.

The concept is similar to Xbox Kinect – a commercial product that uses cameras to recognize gestures – but the UW technology is simpler, cheaper and doesn’t require users to be in the same room as the device they want to control. That’s because Wi-Fi signals can travel through walls and aren’t bound by line-of-sight or sound restrictions.

The UW researchers built a “smart” receiver device that essentially listens to all of the wireless transmissions coming from devices throughout a home, including smartphones, laptops and tablets. A standard Wi-Fi router could be adapted to function as a receiver.

WiSee technology uses multiple antennas to focus on one user to detect the person’s gesture.

When a person moves, there is a slight change in the frequency of the wireless signal. Moving a hand or foot causes the receiver to detect a pattern of changes known as the Doppler frequency shift.

These frequency changes are very small – only several hertz – when compared with Wi-Fi signals that have a 20 megahertz bandwidth and operate at 5 gigahertz. Researchers developed an algorithm to detect these slight shifts. The technology also accounts for gaps in wireless signals when devices aren’t transmitting.

The technology can identify nine different whole-body gestures, ranging from pushing, pulling and punching to full-body bowling. The researchers tested these gestures with five users in a two-bedroom apartment and an office environment. Out of the 900 gestures performed, WiSee accurately classified 94 percent of them.

“This is the first whole-home gesture recognition system that works without either requiring instrumentation of the user with sensors or deploying cameras in every room,” said Qifan Pu, a collaborator and visiting student at the UW.

The system requires one receiver with multiple antennas. Intuitively, each antenna tunes into a specific user’s movements, so as many as five people can move simultaneously in the same residence without confusing the receiver.

WiSee technology uses multiple antennas to focus on one user to detect the person’s gesture.
If a person wants to use the WiSee, she would perform a specific repetition gesture sequence to get access to the receiver. This password concept would also keep the system secure and prevent a neighbor – or hacker – from controlling a device in your home.

Once the wireless receiver locks onto the user, she can perform normal gestures to interact with the devices and appliances in her home. The receiver would be programmed to understand that a specific gesture corresponds to a specific device.

Collaborators Patel and Sidhant Gupta, a doctoral student in computer science and engineering, have worked with Microsoft Research on two similar technologies – SoundWave, which uses sound, and Humantenna, which uses radiation from electrical wires – that both sense whole-body gestures. But WiSee stands apart because it doesn’t require the user to be in the same room as the receiver or the device.

In this way, a smart home could become a reality, allowing you to turn off the oven timer with a simple wave of the hand, or turn on the coffeemaker from your bed.

The researchers plan to look next at the ability to control multiple devices at once. The initial work was funded by the UW department of computer science and engineering.



SOURCE  University of Washington

By 33rd Square

Subscribe to 33rd Square

 Computer Interfaces

A start-up company, Leap Motion have built a device about the size of a cigarette lighter that contains three tiny cameras inside. It attaches to a computer and turns any PC or Mac into a gesture-recognition device. The idea is similar to the one behind Microsoft’s Kinect, the device that lets people play games just by moving their hands and body.

Where Microsoft's Kinect brought the idea of 3D motion tracking to mainstream adoption in the marketplace, the 3D camera's low, 640x480 resolution, moderate input lag, and bulky form factor have somewhat limited its use cases.

Now, a new startup called Leap Motion wants to revolutionize the motion-tracking space with a chewing-gum-pack-sized 3D sensor that promises sub-millimeter accuracy.

Leap Motion claims its small sensor creates a "three-dimensional interaction space" of four cubic meters that can track the 3D position and orientation of individual fingers and even thin objects like pencils in real time, all to a tolerance of one-hundredth of a millimeter.

The company says the breakthrough in resolution comes not from the hardware, which consists of relatively standard parts, but from what CTO David Holz calls "a number of major algorithmic and mathematical problems that had not been solved or were considered unsolvable."


In the video demo below prepared for Cult of Mac, Holz demonstrates what appear to be very responsive and accurate gestures with his finger tips and hands.  Some may argue that as for replacing the mouse, the Leap Motion interface might not be comfortable over long stretches, however with the kind of performance exhibited in the video, the device is intriguing.  Moreover, the Kinect has been very successful for Microsoft and is being extended into a number of uses, including robotics.  If Leap Motion can do the same via its developer program, it may be an ideal device for robotic sensing and much more.

The San Francisco-based company has just started taking preorders for the $69.99 PC- and Mac-compatible device, which it plans to ship in December. It's unclear exactly what apps the system will support at that time, but the company told CNET it has plans to offer up 15,000 to 20,000 free development kits to kick-start the software market on its open platform.






SOURCE  Core 77

By 33rd Square

Subscribe to 33rd Square

In the video below a demonstration of NAO movement by using Kinect by NAO Developer Zecloud and Taylor Veltrop from Aldebaran Robotics at the Microsoft Tech Days, Paris on 7th February 2012.  Not quite as impressive as the cat grooming demo, but impressive nevertheless.