Invited Paper AIM 2001(IEEE/ASME International Conference on Advanced Intelligent Mechatronics) Teatro Sociale, Como, Italy July 2001 Robotics and Machine Vision for the Future  -- an Industrial View -- Masakazu Ejiri, Fellow, IEEE, IAPR, and IEICE Central Research Laboratory, Hitachi, Ltd., Kokubunji, Tokyo 185-8601, Japan Abstract Recent trends in industrial technology are to make things small, synergetic, intelligent, and environmentally friendly. Mechatronics is one research area on these trends, and its perspective is first introduced. The status of research in robotics and machine-vision technologies is then described as a typical example of mechatronics research. Expectations for the future of these technologies are also mentioned from the viewpoint of industry, emphasizing the importance of considering the reliability in robotics and of studying real-time color video processing in machine vision. These fields are becoming increasingly important for establishing a productive, efficient, secure, and stress-free society through factory, office, and social automation. 1 Introduction Research on mechatronics has made great progress over the last 30 years. In particular, industrial robots and automatic machines equipped with machine vision have played an important role in our society and have contributed significantly in making it productive and efficient. In this paper, we briefly discuss the present status of robotics and machine vision technologies from the industrial viewpoint and suggest some directions of these fields in the future. 2 Trends in Industrial Technologies Industrial technologies and products are being always innovated. And to improve their functions and performances, novel materials, parts, mechanisms, architectures, and algorithms are continuously being put into use. Higher precision, higher resolution, and higher speed are some examples of targets being aimed at. These targets form the main trend in the technology development underlying the daily activities, especially in the manufacturing industry. By observing the history of the technology over the last 30 years, we can see four more trends in addition to this fundamental one. The first trend is to make things smaller. Downsizing in computer systems has made it possible to distribute computing power everywhere. In this background, there have always been drastic improvements in semiconductor technology. Size reduction, power reduction, and scale increase in integration are the main efforts made in the development of electronic parts. As a result, a large number of mobile phones and personal data assistants have also become available. By taking advantage of semiconductor technology, the effort to make mechanical parts smaller has also started, forming a new research area called micro machines.  The second trend is to combine things for synergism. Variously advanced technologies are integrated or fused in order to achieve novel technologies and machines. The fusion of different technologies usually necessitates another technology that behaves as a catalyst or an adhesive. Mechatronics is one typical example of the fusion between mechanical engineering and electronic engineering, to which automatic control technology serves as the adhesive. In recent years, network technology is becoming a powerful adhesive to create new technologies. The third trend is to make things more intelligent. The end users of most products are humans; thus, the products must have a friendly interface so they are easy to use without any perplexity. As the visual function is the most prominent function in humans, machines with image-based interfaces and multimedia handling capabilities are becoming more and more important. The forth trend is to make things more environmentally friendly. Without harmony with the environment, no products will be validated from the viewpoint of product liability. It is thus being urged nowadays to employ designs that are easy to disassemble and destroy and to use materials and parts that are easy to reuse.  These four trends in addition to the main underlying one will continue in the future; as a result, low-cost, high-performance, highly-functional, meaningful and tender products will be expected to appear.   3 Perspective of Mechatronic Products Classical machines such as the windmills and clocks of centuries ago were purely mechanical; they used cams, cranks, and gears for changing the motive power to other modes of motion. With the lapse of time, more complicated and sophisticated motion became attainable by the combination of rather simple mechanisms each controlled at first electrically and afterwards electronically. In the 1970s, the importance of the boundary between mechanical and electronic engineering was recognized more clearly among researchers by being encouraged with the improvement of microcomputer and memory technologies. The compound word “mechatronics” was thus formed by enthusiastic Japanese researchers in control field, and was proposed to best describe this new area of the research.  In addition to industrial robots, typical examples of mechatronic products are facsimiles, printers, video cameras, magnetic storage devices, optical storage devices, automatic teller machines, automatic ticketing machines, and automatic mail-sorting machines. A common feature of these products is the handling of thin and weak objects such as paper sheets and plastic tapes/sheets/disks. Handling of these objects is not easy or is almost impossible without using sophisticated electronic control.  Another common feature of the mechatronic products is that they handle some sort of information. Massive amounts of data received are correctly recorded, are re-constructed as documents and pictures, or are analyzed and used as the information for classification of objects. In other words, dense data exchange with other electronic devices or with the outside world is essential in most of those products.  Mechatronics has thus been an inevitable solution to the problems that electronics or mechanics alone could not solve. And nowadays, mechatronics can further be recognized as a concept representing “electronically controlled information-handling machines”, in which the electronic, control, information, and mechanical engineering are tightly integrated. In terms of price-to-weight ratio and the ratio of electronic to mechanical parts, some existing mechatronic products are plotted in Fig. 1 to clarify their relative positions among other industrial products [1]. The mechatronic products have a fairly balanced number of electronic parts and mechanical parts, whose ratio roughly ranges from 0.5 to 2.0. However, as semiconductor technology progresses, the number of electronic parts is apt to be reduced through higher integration, so the ratio will shift to the more mechanical direction. Furthermore, since it is easier to reduce the size and add value to electronic parts compared to mechanical ones, mechatronic products will be shifted in the electronic direction. Thus, the ratio of the number of parts is merely a reference index and does not simply represent the degree of how electronic or how mechanical the product is.  As shown in Fig.1, the price of all industrial products normalized by their own weight ranges between 1-1000 yen/gram (0.3-300 dollars/ounce). For example, both an office-use large-scale laser printer and home-use tiny ink-jet printer cost 10-20 yen/gram. Of course, products with a long history, high maturity, high popularity, and keen competition are apt to have a lower price per unit-weight. It is also interesting to note that, with exception of precious metals, artistic works, antique objects with a scarcity value, and recent computer software, data, and information contents, all industrial products that humankind can create to date range between the price of gold and grain (rice or wheat). In other words, our industrial activity has not produced anything more valuable than gold. 4 Robotics Technology 4.1 Robotics in Japan Industrial robots are the typical mechatronic products to which, in our knowledge, the word “mechatronics” first began to be used. It has been more than three decades since robots turned from being fictional creatures to engineering reality; thus, mechatronics has a history of more or less 30 years. The Japan’s first introductory paper on the robotics technology appeared in an academic journal in 1963. Five years later, two types of early industrial robots were imported from the USA. And two years later, in 1970, Japan’s original intelligent robots with computer-controlled vision systems were developed at two different laboratories and were demonstrated successfully. Since then, industry became aware of and interested in these novel technologies. So drastic efforts to create industrial robots began; as a result, the technology attained to its peak. For example, horizontally articulated high-precision assembly robots (SCARA-type, which stands for Selective Compliance Assembly Robot Arm) were invented and have gained worldwide popularity. Also, the enthusiasm of industry leaders to utilize robots in their factories made Japan one of the most advanced industrial societies by installing and operating, at present, more than 400,000 industrial robots in total. 4.2 Industrial Robots Advances in the technology of industrial robots, however, seem to have stagnated in recent years. It is thus clearly the time to start new investigations to break through this stagnant situation in the present robotics industry. In addition to the continuous efforts to achieve higher speed, higher precision, lower cost, safer, and more reliable industrial robots, young robotics researchers face three challenges [2]. The first challenge is to shift attention from manufacturing to design processes. By attempting to effectively automate whole “production” processes, automation must be expanded from the product-manufacturing process to the more upstream processes; i.e., to product realization and further to product incubation. Total production efficiency will not be attained if robotics technology is restricted within the manufacturing process. Future robotics researchers are expected to play an important role in the automation of product incubation and realization processes in order to facilitate the smooth transition of design knowledge among these processes, thus achieving truly efficient production.  The second challenge is to shift attention from single to multiple capabilities. With the progress of “individualization and sensuousness”, the variety of product types each being added with a different value may be required to appear. Production methods will thus be affected, and some factories are again starting to adopt a traditional assembly method, by which a worker is responsible for assembling the whole product from beginning to end. This means the return to a multi-functional operation from a distributed single operation used in conveyor-type assembly. Therefore, more flexible multi-functional robots may be needed as cooperative assistants to human workers. In these robots, a skill in performing a single task must be combined with other skills in different tasks in order to perform a complex operation, in which tasks are so dexterously and sometimes concurrently performed that no single task is separately noticeable. Research on skill acquisition of robots by experience-based self-learning and research on giving the learning incentives to robots are quite difficult but will be rewarding in the future.  The third challenge is to shift attention from assembly to disassembly. Applications of robots to the disassembly of the used products for recycling seem to be important for the future. The concept of an adaptive robot for ecologically conscious industry has been proposed. It reads the identification number and the manufacturer of wasted products from a tag attached in advance when they were produced, and retrieves the product knowledge via a network so it can decide on the optimum disassembly sequence. In disassembly applications, objects can probably be handled quite differently. Robots may be able to throw objects or scrape up the destructed parts for efficient handling. Thus, a new technological paradigm may arise. 4.3 Autonomous Robots Although recent research on autonomous robots is fairly active, what is missing seems to be a clear vision of possible applications and attention to reliability and power-related issues [1]. The control technology for legged robots has greatly progressed in recent years. However, the researchers are apt to focus only on information-related issues and energy-related issues are sometimes ignored. Thus, prototype intelligent robots often have long power-supply trails or carry heavy batteries on their backs, which last only 30 minutes or so, and have to rest for a couple of hours for re-charging. The past tendency of the power density increase in batteries is only twice in ten years, so it may take 50 more years to develop the right types of batteries for autonomous robots with appropriate energy density, size, and weight. However, the recent developmental efforts of battery-driven cars have given hope that the inconveniences of batteries will be overcome. Robotics researchers must also pay more attention to this problem and not merely wait until someone else solves it. In immediate applications, however, a systems-engineering approach may give an alternative solution to this problem until the right types of batteries are developed.  Symbiotic robot usage may become more serious in a society envisioned by some researchers; namely one where robots wander around the street. However, this is not the type of society we are aiming for. Robots operating in the street should probably be made smaller, lighter, and less speedy than humans for the safety reasons. Then, what can these robots do for humans? Carrying objects for humans may probably be too slow to be realistic, and carrying information for transmitting to and from humans can probably be done by electronic post at street corners by means of wireless IC cards or wristwatch type novel electronic devices. Recently, two-legged robots developed by Japanese companies have drawn considerable attention from the public. Also, a variety of animal-like pet robots with some sensor-based learning capabilities are being marketed. These products are fresh breezes to blow through the stagnant robotics industry, and they are welcome in the sense that such mind-stimulating robots will reactivate the robotics industry. The hidden problem is, however, the lack of future perspectives on the effects of these robots. A society in which the stimulation of the human mind can only be achieved by such artifacts will not be a sound society. We have to note that the artifacts also have a life span and will die. Too much emotion and reliance on these robotic artifacts may pose another unexpected problem in the future.  Autonomous robots are in fact very interesting not only for technologically innocent people but also for engineers. Many robot applications can be conceived for the use in the power industry, in the construction industry, and in disaster prevention and rescue. Also, autonomous robots may be helpful in understanding human functions and behavior. Especially in the educational environment, prototyping of an autonomous robot will provide dreams for young students as well as the joy of learning mechanisms and creating new things. However, one anxiety is that this may bring up “day-dreamers”. Their prototype robots may only work once and may not have to work again once they are taped on video. That is, education based on reliability seems to be missing. Reliability is a very important concept for every industrial activity, especially for design work, which the students may engage in after graduation.   5 Machine Vision Technology  5.1 Factory Applications Machine vision is an important function that future robots must have. One of the first successful applications of machine-vision technology was to the automatic assembly of transistors in 1973 [3]. By using local-parallel-type image processor as a means of real-time pattern matching, two-dimensional positions of transistor chips in microscope images were automatically determined for wire bonding. Later on, this application was expanded to the assembly of ICs and LSIs and also to the inspection of semiconductor wafer patterns and printed circuit boards [3][4]. Consequently, the assembly and inspection of electronic parts became a significant market for the machine vision.  Meanwhile, the performance of microprocessors improved greatly and the capacity of memory chips increased drastically. Also, special-purpose image-processing LSIs were developed and became available. These all facilitated the use of gray-level images instead of binary ones until then, and contributed to production of general-purpose, programmable machine-vision systems. Such systems are now being used in many industrial sectors such as electronics, machinery, medicine, and the food industry for the automation of manufacturing processes [5][6].  In brief, the machine-vision systems used in factory automation (FA) were aimed at realizing a productive society, and progress was made in such applications as: ・　Position detection for assembly, ・　Shape detection for classification, and ・　Defect detection for inspection. Though these machine-vision systems do not look like robots, it is our belief that they can also be called robots, because their aim is to realize automatic execution of tasks that had, until then, only been done by skilled human workers. 5.2 Office Applications In addition to the FA, applications of machine-vision technology to office automation (OA) have also progressed. For example, by fusing the data from optical and magnetic sensors, automatic recognition of monetary bills was achieved and led to automated teller machines (ATM) for banks. The development of technologies to analyze drawings, maps, and documents followed. The resulting geographic information systems (GIS) and document readers are now being utilized for facility management at public service companies, for customer management at banks, and for crisis management and disaster simulation at local governments. By using newly developed mail-sorting machines, postal automation systems are now being renovated. Such systems can automatically read hand-written Kanji addresses and convert them to transparent barcodes, which are later used for mail assembly in the order of the houses on delivery route [7]. In short, machine-vision systems were used for OA with the aim of realizing an efficient society, and progress has been made in such fields as: ・　Efficient handling of large-scale data, and ・　High-precision and high-speed recognition and handling for paper-based information. In other words, OA has been a typical application field of not only machine vision but also other mechatronic products, and it will continue to be one in the future. The characteristics of the image processing being utilized in machine-vision applications are illustrated in Fig. 2. In addition to FA and OA, the next important target of application seems to be SA, i.e., social automation.   5.3 Social Applications In general, the things that move in our society are humans, physical objects, money, and information. The mobility of the last two is becoming fairly smooth, quick and efficient, thanks to the progress of network technology.  The mobility of humans is realized by traffic systems, and traffic control is a key technology for achieving a smooth flow of humans. The first application of machine vision to traffic use was probably the elevator-eye system in 1977 [7], At that time, however, the number of systems installed was very limited, so the business was not too successful. In this system, a video camera installed in each elevator hall recognized the number of persons waiting for the elevator at each floor, and designated an elevator to quickly serve the crowded floor. Nowadays, many studies are being done in the field of car traffic control, where the recognition of license number plates, traffic flows, the degree of congestion, and illegally parked cars, etc. is being tried. These approaches will probably be combined as an important function of the future intelligent transport system (ITS), which is a typical example of SA. Physical distribution systems control the flow of physical objects. They usually identify and dispatch a variety of objects by the help of barcode systems. Conventional machine-vision systems are mainly for shape and position recognition, and will not simply solve the identification problem especially in large-scale physical distribution systems. Instead, as an extension or substitution of barcodes, a novel machine vision combined with wireless semiconductor-chip type “information carriers” may probably play an important role in future physical distribution systems.  As a part of SA, machine-vision applications will also realize a more comfortable and secure human life. For example, a river-water monitoring system using fish behavior, an intruder-monitoring system based on image-change detection at railroad crossing, and a remote monitoring system of volcanic activities have been developed. A GIS-based restoration-assistance system after a disaster such as an earthquake and an intelligent on-line monitoring system during medical surgery have also been developed and are being planned for further improvement. As indicated in these examples, applications of machine-vision technology to human welfare, medicine, and environment improvements are very promising.  The key concept representing the future seems to be the realization of a calm society; that is one in which security will be guaranteed and all uneasiness will be relieved through SA, more specifically, through “networked” social automation. The SA is a synonym of “networking technology” in one sense, and the “networked machine vision” may become an important key technology for realizing SA. In this sense, the most important objectives of machine vision may eventually be converged to the realization of the following two functions: ・　24-hour/day abnormality monitoring via networks, and ・　Personal identification via networks. To date, motion capturing, face recognition, facial expression understanding, gesture recognition, sign language understanding, and behavior understanding are keen interests among researchers. The progress of machine-vision technology to date and in the future is summarized in Table 1.   5.4 Dynamic Video Processing In most SA applications, dynamic image processing, which analyzes and/or synthesizes color-video images in real-time, will be the key to success [7]. The technology already developed includes real-time scene-separation based on an algorithm for quickly finding changes between consecutive image frames [8]. This technology not only forms the basis of video editing but also forms the basis of abnormality finding. For example, a broadcasting company has put it into use for video inspection so that subliminal advertising can be detected before a video goes on the air.  One interesting example of dynamic video analysis and synthesis is “Tour into the picture (TIP)” technology. A 2-D picture is scanned into a computer and is interpreted as a three-dimensional model by fitting vanishing lines on its display. The picture can then be looked at from different angles and distances [9], thus generating a motion video from a single picture. For example, viewers can feel as if they were taking a walk in an ancient city when an old picture of the city is available. Another dynamic image processing technology is called “Cyber BUNRAKU”, in which human facial expressions are recognized by small infrared-sensitive reflectors put on a performer’s face. By combining the facial expressions thus obtained with the limb motions of a “Bunraku doll”, which is usually used in traditional Japanese theatrical performance, a 3-D character model in a computer can be choreographed in real-time to create video images [10]. This technology will be one of the fastest and most effective ways to produce multimedia animation programs. These simple methods mentioned above can combine real images and computer graphics together and, thus, can easily create video contents for use in mechatronic products. For example, an ATM can be equipped with animation-based sign-language instructions, and a rehabilitation machine for walking-disabled persons can be fitted with a specific video for both instruction and enjoyment during the tedious training period.   6 Conclusion An industrial view on robotics and machine-vision technologies, together with that on mechatronics, was briefly introduced. Expectations for the future of these technological fields were also mentioned, emphasizing the importance of the reliability consideration in robotics and the importance of studying real-time color video processing for novel machine vision. It was concluded that these fields would be important in establishing a productive, an efficient, as well as a calm society through networked social automation.   References [1] Ejiri, M.: “Towards meaningful robotics for the future; Are we headed in the right direction?”, Robotics and Automation Systems, 18, pp.1-5 (1996) [2] Ejiri, M: “Expectations of new robotic technology from an industrial viewpoint”, Journal of RSJ, 16, 1, pp.39-40 (1998) (in Japanese) [3] Ejiri, M.: “Machine vision: A practical technology for advanced image processing”, Gordon & Breach Sci. Pub., New York (1989) [4] Ejiri, M., Yoda, H., and Sakou, H.: “Knowledge-directed inspection for complex multi-layered patterns”, Machine Vision and Applications, 2, pp.155-166 (1989)  [5] Ejiri, M.: “Recent image processing applications in industry”, Proc. 9th SCIA, Uppsala, pp. 1-13 (1995) [6] Ejiri, M.: “Machine vision: A key technology for flexible automation”, Proc. of Japan-U.S.A. Symposium on Flexible Automation, Otsu, Japan, pp. 437-442 (1998)  [7] Ejiri, M., et al.: “Evolution of real-time image processing in practical applications”, Proc. IAPR MVA, Tokyo, pp. 177-186 (2000) [8] Nagasaka, A., Miyatake, T., and Ueda, H.: “Video retrieval method using a sequence of representative images in a scene”, Proc. IAPR MVA, Kawasaki, pp. 79-82 (1994) [9] Horry, Y., Anjyo, K., and Arai, K.: “Tour into the picture: Using a spidery mesh interface to make animation from a single image”, Proc. ACM SIGGRAPH 97, pp. 225-232 (1997) [10] Arai, K. and Sakamoto, H.: “Real-time animation of the upper half of the body using a facial expression tracker and an articulated input device”, Research Report 96-CG-83, Information Processing Society of Japan, 96, 125, pp. 1-6 (1996) (in Japanese)