Keystone distortion brings trapezoidal distortion in an image plane, which causes both vertical and horizontal parallax in the stereoscopic image (Kang and Lee2009).
Although several solutions for automatic keystone correction have been proposed, these approaches typically only deal with stationary image and do not work for real-time continuous processing of data from a moving camera. The aim of this paper is to introduce a novel approach for evaluating rotation angle using keystone dis- tortion information in real-time. To achieve distortion in real-time, the proposed RGBD-camera system isfirst calibrated offline, then uses the depth images from the RGBD camera and compute it’s relative pose with respect to the camera. Finally remove the keystone distortion by compensating the discrepancy of relative orientation between the camera and the image plane (Petrozzo and Singer 2000).
Fig. 2 QR code structure:aoverview,bfinder pattern,cnotation used for QR code extraction method (Ahn and Lee2014)
Mobile Robot Applied to QR Landmark Localization… 49
The proposed method consider distances between camera and left/right edge of the QR landmark is to estimate the keystone distortion angle. The method and the geometrical transformation is then used to evaluate the rotation angle of QR landmark in an image plane.1
Let us denote the left and right edge byh1andh2respectively (Fig.3), andd1 and d2are the distances between left and right edge and the camera plan. In this work (Fig.3a), it is shown that the edgeh1is shorter and edgeh2is longer due to the rotation of the image plane. Using these edgesh1andh2, estimate the distances between left/right edge of the QR code and the camera plan in order to obtain the rotation angle c1. Table 1 illustrates the experimental analysis of estimating dis- tances (d1,d2) between the left/right edge of QR code and the camera plane at a distance 100, 150 and 200 cm respectively (Dutta2015). Whered= Length of QR code (Fig.3b). Using distances d1and d2, it is possible to evaluate the rotation angle (c1).
Fig. 3 Geometric model of estimating distortion angle based on keystone effect
Table 1 Distance estimating depends on rotation angle
Angle Distance at 100 cm Distance at 150 cm Distance at 200 cm d1(cm) d2(cm) d1(cm) d2(cm) d1(cm) d2(cm)
10° 102 98.2 151.2 148.9 201.2 198.9
15° 103.5 97.8 151.9 148.3 201.9 198.3
20° 104.2 97 152.5 148 202.5 198
25° 105 96.6 152.8 147.6 202.8 197.6
30° 105.3 96.1 153.1 147 203.1 197
35° 106.5 95.4 153.6 146.4 203.6 196.4
40° 107 95.9 154 146 204 196
45° 107.8 94.3 154.4 145.7 204.4 195.7
50° 108.2 93.6 155 145.1 205 195.1
1The idea about the geometrical derivation has been taken from article (Dutta2015).
50 V. Dutta
aẳarccosd12ỵd22d2
2:d1:d2 ð6ị
Similarly, calculate the angleb. Then,cis expressed by,
cẳ180ab ð7ị
Thereafter, it is obtainedb1andb2(as shown in Fig.3b1=b2). Using Eq.7,b1can be evaluated as,
b1ẳ180a
2 ð8ị
Finally,b1and bare determined, thereforec1can be expressed by,
c1ẳbb1 ð9ị
Based on the above calculation, the rotation angle (c1) is evaluate, which often appears virtually twisted in an image plane due to keystone distortion.
4 The Necessity of QR Landmark Localization for Mobile Robot Applications
QR landmark localization allows a robot to perform an action in a visual landmarks framework, where a robot is expected to execute a given task. In our experiment, the QR landmarks provide semantic information, which can help to overcome the complexities and limitation of recognition and assist the scene understanding.
Semantic knowledge extended by the QR landmarks is obtained in human form of representations in the environment and the robot’s actions are studied to show the context-based robot actions practicability.
Aiming at a mobile robot action task, an environment cognition method named as QR landmark-based robotic actions is proposed. The semantic planning approximates the human point of view of robot environments, which enable high-level and more intelligent robot development (Wu et al.2014). However, the QR code has the merits of storing object’s name, other attribute information, high performance of keeping secret and anti-counterfeiting, etc.2
The robot reads the QR code and decodes the information immediately in the observed image and establishes a correlation between the observed QR landmark’s (i.e., node) and the corresponding robot’s relative position (i.e., odometry position).
This foundation guarantees the management of environments (see Fig.4). Thereby
2Afirst version of this work was published in“10th Young Scientists Conference”, Warsaw, Poland, September 21–23, 2015.
Mobile Robot Applied to QR Landmark Localization… 51
the robot can determine the relation between the QR landmark and the robot relative position in the environment. The author’s apply the following formula to the visual landmarks scenario for mobile robot actions,
SOi ẳðO1;O2;O3;. . .;Onị ð10ị
where
Oi2V ð11ị
OiẳOiðgị ð12ị
vj2V; vjẳvjðw;gị ð13ị
Sis the set of objects,Oiis the set of object’s information,Vis the set of marked QR codes in the environment,vjis the number of QR code marked objects,gis the specific property of an object,wis the QR landmark’s position in the environment.
5 Mobile Robot Actions Based on QR Landmarks
The action of robot in a visual landmark scenario, a path from the robot’s relative position to a considerable QR landmark has to be known that they are considered to be reachable in a given time.
Figure5 illustrates the geometrical representation of robot’s relative position with QR landmark. Assuming, (xR,yR) is the position within the world reference Fig. 4 Environment with QR landmarks
52 V. Dutta
frame in thexyplane andhRis the orientation with respect to the x-axis,h2 ẵ0;2p (Dutta and Kesswani2014).
In this work, the author’s introduce a minimum distancedmin= 0.3 m, where the robot must stop once it reaches the destination. The proposed algorithm is used to perform the given task in complex indoor laboratory environment with landmarks in different areas, assuming, knowledge of robot’s relative position in the envi- ronment from the odometry. Upon using robot’s relative position, the QR landmark is mapped in the given environment. If (xL, yL) is presumed to be landmark’s location in the environment and (xR,yR) is the robot’s current position, then,
xLẳxRỵ ðsinh:Ddị ð14ị yLẳyRỵ ðcosh:Ddị ð15ị
Computer vision is used to measure the distance assuming that the size of the QR code is fixed. From the size of the QR landmark seen in the image, the distance Ddbetween the robot and the QR landmark is obtained.
DdẳfRh
Ih
ð16ị
where f is the focal length (cm),Rhis the real height of the QR code (cm), andIhis the QR code height in the image (pixels). However, in this work, the proportional size of the QR landmark depends on the distances relative to the robot (see Fig.6).
Figure6illustrates the images taken by the camera mounted on the robot during the real-time distance measurement (Fig.7).
Fig. 5 Geometrical representation of robot position with QR landmark
Mobile Robot Applied to QR Landmark Localization… 53
Fig. 6 Image of QR code observed at a distance:a1.5 m,b1.0 m,c0.7 m,d0.5 m
54 V. Dutta
The proposed algorithm improves the efficiency in landmark recognition with obtaining the information about the objects, whichfinally speeds up in mobile robot action tasks.
6 Experimental Works and Evaluation
The author’s tested 45 samples of QR Code to evaluate the performance of the proposed method. This experiment is carried out under ROS (Robot Operating System) with vision sensor (kinect), using Zbar-library (Brown2014) to process the QR landmarks for both encoding and decoding.
Table2 presents the results of angular error testing for the landmark’s position with improvement of keystone distortion for distances (di) equal to 100, 150 and 200 cm respectively.
Fig. 7 Image of QR code observed at an angle:a15°,b18°
Table 2 Average angular
errors without keystone effect Code type Angle Orientation error (c1) d100 d150 d200
QR 10° 0.3° 0.3° 0.5°
QR 15° 0.3° 0.8° 0.5°
QR 20° 0.5° 0.6° 0.8°
QR 25° 0.4° 0.9° 0.9°
QR 30° 0.55° 0.7° 1.12°
QR 35° 0.5° 0.9° 1.12°
QR 40° 0.45° 0.8° 1.17°
QR 45° 0.7° 1.0° 1.3°
QR 50° 0.6° 0.7° 1.1°
QR 55° – – –
Mobile Robot Applied to QR Landmark Localization… 55
Variety range of QR landmarks are collected to evaluate the proposed algorithm on mobile robot action tasks based on visual landmarks framework and an exper- iment is carried out in the laboratory environment using Seekur Jr mobile robot (LLC2014). For each action, the localization duration, landmark’s position (based on the robot’s relative position) and distance has been recorded. Figure 8illustrates average angular error and distance measurement error during experiment at dis- tances (50, 100, 150, 200, 270 cm).
The experimental results are expressed in terms of robot trajectory to reach the destination (Fig.10). The distance to reachable way points during the mobile robot action is a parameter as important as the duration of the localization of the QR Fig. 8 Result’s graph of estimating error in angular orientation and distance measurement based on proposed method.aResult’s graph of average orientation error.bResult’s graph of average distance error
56 V. Dutta
landmark. Indeed if the robot moves far away from the QR landmark reachable points, it increases the risk of collision and falls. To evaluate moves his distance, the actual best path (straight path from robot’s current position to QR landmark) followed by the robot is computed. Figures8 and 9 shows that, the proposed method not only minimizes the angular error but also speeds up in the navigation while mapping the landmark’s location in the environment.
The experiment is carried out in an imitation domestic environment in our laboratory. The whole process of mobile robot actions and mapping all the QR landmarks are illustrated in Fig.11.
Fig. 9 Navigation duration during the indoor mobile robot action tasks based on the proposed method.aResult’s graph of duration with 5 objects during mobile robot actions.bResult’s graph of duration with 8 objects during mobile robot actions
Mobile Robot Applied to QR Landmark Localization… 57
7 Conclusion
This paper introduced the method of QR code localization with evaluating the virtual rotation towards the image plan and the distance to the mobile robot. In this paper, localization is performed with a fully calibrated camera. During the exper- iment, it was estimated that the maximum angle and the distance from which the Fig. 10 Robot’s path during mobile robot actions.Blackand green arrowsindicate the mean position and orientation of the QR landmarks stuck on the objects. Green circle defines the observation point. Black line with green arrows indicates the mobile robot actions without estimating keystone distortion angle, where asblue line with arrowsdefines the robot trajectory in mobile robot actions using proposed method. All units are in meters
Fig. 11 The whole process of mobile robot actions and mapping QR landmark’s location in a given environment
58 V. Dutta
robot can see the QR code are respectively 51° and 270 cm. The experiment with mobile robot in Table2 and Figs.8 and 9 confirm reliable localization of QR landmark within the above-stated limits. It is shown that, being able to infer the Keystone Effect (KE) enables the robot to perform the tasks in a more meaningful way. In extensive experiments over a challenging scenario, the results indicate that through the proposed method, QR landmark can be mapped by the mobile robot and serial landmarks can facilitate robot actions in a complex area. The experi- mental results shows in Fig.11 is just an one scenario. The experiment with the proposed method has been carried out in different possible complex scenarios and the results shows the improvement in indoor environments.
Acknowledgments This work was supported by the HERITAGE project (Erasmus Mundus Action 2 Strand 1 Lot 11, EAECA/42/11) funded by the European Commission. The author gratefully acknowledge Prof. Teresa Zielinska for supervising throughout the research and provide all necessary help and resources.
References
Ahn, M., & Lee, S. (2014). A research on QR code recognition enhancement using pre-constructed image matching scheme. In 2014 International Conference on Information and Communication Technology Convergence (ICTC) (pp. 82–83). IEEE.
Aleksandrovich, Y. D., Gennadievich, P. G., Stepanovich, K. A., & Zalmanovich, M. V. (2013).
Mobile robot navigation based on artificial landmarks with machine vision system. World Applied Sciences Journal,24(11), 1467–1472.
Belussi, L. F., & Hirata, N. S. (2013). Fast component-based QR code detection in arbitrarily acquired images.Journal of Mathematical Imaging and Vision, 45(3), 277–292.
Bernsen, J. (1986). Dynamic thresholding of grey-level images. InInternational conference on pattern recognition(pp. 1251–1255).
Bkaczyk, R., & Kasinski, A. (2010). Visual simultaneous localization and map-building supported by structured landmarks.International Journal of Applied Mathematics and Computer Science, 20(2), 281–293.
Borkowski, A., Siemiatkowska, B., & Szklarski, J. (2010). Towards semantic navigation in mobile robotics. In Graph transformations and model-driven engineering (pp. 719–748). Berlin:
Springer.
Brown, J. (2014). Zbar Software.http://zbar.sourceforge.net/. (Online) Accessed November 19, 2014.
Dutta, V. (2015). QR code localization using keystone effect. InYoung scientists towards the challenges of modern technology, September (pp. 21–23).
Dutta, V., & Kesswani, N. (2014). Designing vision based autonomous docile-x mobile robot for real-time application to soccer behaviors. Journal of Automation Mobile Robotics and Intelligent Systems, 8(4), 40–50.
Ismail, A., Jamil, S., Ismail, A. H., Ayob, M., & Rahim, N. A. (2012). A comprehensive study of using 2d barcode for multi robot labelling and communication. International Journal on Advanced Science, Engineering and Information Technology, 2(1), 80–84.
Kang, W., & Lee, S. (2009). Horizontal parallax distortion correction method in toed-in camera with wide-angle lens. In3DTV conference: The true vision-capture, transmission and display of 3D video(pp. 1–4). IEEE.
Mobile Robot Applied to QR Landmark Localization… 59
Lee, Y.-C., Chae, H., Kim, S.-H., et al. (2011). Applications of robot navigation based on artificial landmark in large scale public space. In 2011 IEEE international conference on robotics and biomimetics (ROBIO) (pp. 721–726). IEEE.
Li, W., Duan, F., Chen, B., Yuan, J., Tan, J. T. C., & Xu, B. (2012). Mobile robot action based on QR code identification. In2012 IEEE international conference on robotics and biomimetics (ROBIO)(pp. 860–865). IEEE.
Li, Z., Wong, K.-H., Gong, Y., & Chang, M.-Y. (2011). An effective method for movable projector keystone correction.IEEE Transactions on Multimedia, 13(1), 155–160.
LLC, A. M. (2014). Seekur Jr mobile robot. http://www.mobilerobots.com/ResearchRobots/
SeekurJr.aspx. (Online) Accessed March 03, 2014.
Petrozzo, R. A., & Singer, S. W. (2000). Cinema projection distortion.SMPTE Journal, 109(8), 648–651.
Rostkowska, M., & Topolski, M. (2015). On the application of QR codes for robust self-localization of mobile robots in various application scenarios. InProgress in automation, robotics and measuring techniques(pp. 243–252). Berlin: Springer.
Skrzypczyński, P. (2009). Simultaneous localization and mapping: A feature-based probabilistic approach. International Journal of Applied Mathematics and Computer Science, 19(4), 575–588.
Wang, D., Jia, Y., & Cao, Z. (2009). A Unique Multi-functional Landmark for Autonomous Navigation. In 2009 Fifth International Conference on Natural Computation (ICNC'09) (Vol. 5). IEEE.
Wu, H., Tian, G. H., Li, Y., Zhou, F. Y., & Duan, P. (2014). Spatial semantic hybrid map building and application of mobile service robot.Robotics and Autonomous Systems,62(6), 923–941.
YANG, S., & SHANG, Z. H. (2008). A New Binarization Algorithm for 2D Bar Code Image [J].
Journal of Kunming University of Science and Technology (Science and Technology),1, 011.
60 V. Dutta
A Collective Behaviour Framework for Multi-agent Systems
Mehmet Serdar Gỹzel and Hakan Kayakửkỹ
Abstract This paper addresses a novel framework that employs a decentralized strategy for collective behaviours of multi-agent systems. The framework proposes a new aggregation behaviour that focusses on letting agents on the swarm agree on attending a group and allocating a leader for each group. As the leader starts moving towards a specific goal in a particularly cluttered environment, other members are enabled to move while keeping themselves coordinated with the leader and the centre of gravity of the group.
Keywords Multi-agent systems Decentralized architecture Collective
behaviour Swarm intelligence
1 Introduction
Swarm robotics is a scientific discipline to collective robotics, inspired from the behaviours of social animals. Multi-agent systems, considered to be aggregations of autonomous agents, resembles swarm robotics concept in a certain way (Brambilla et al.2013). Accordingly, both concepts will be considered together in this study.
Collective behaviours of multi-agent systems can be classified into three main groups namely, collective decision making, navigation behaviours and spatially organizing behaviours (Brambilla et al. 2013). Aggregation is one of the funda- mental and critical spatial organization that allows a group of robots to get close one other, providing interaction and collective movements (Camazine et al.2001).
Aggregation behaviour can be observed in nature frequently, such as bacteria, bees,fish and etc. (Camazine et al. 2001; Jeanson et al.2005). Probabilisticfinite state machines (PFSMs) are the main methodology used in aggregation ensuring thatfinally only a sole aggregate is formed. Each robot starts exploring the envi- ronment so as tofind other robots. Once other robots are found, it decides whether
M.S. Gỹzel (&)H. Kayakửkỹ
Computer Engineering Department, Ankara University, Ankara, Turkey e-mail: mguzel@ankara.edu.tr
©Springer International Publishing Switzerland 2017
D. Zhang and B. Wei (eds.),Mechatronics and Robotics Engineering for Advanced and Intelligent Manufacturing, Lecture Notes
in Mechanical Engineering, DOI 10.1007/978-3-319-33581-0_5
61
to join or leave the aggregate in a stochastic manner (Garnier et al.2005; Soysal and Şahin2005,2007). Alternatively, artificial evolution approach has been employed to automatically select aggregation behaviour (Soysal et al. 2007). Coordinated motion, flocking, is a navigation behaviour inspired from fish or flock of birds (Kaminka et al.2008). In multi-agent system, coordinated motion approach pro- vides safer navigation for a group of robots while keeping a constant distance from one another based on virtual physic-based design. One of the popular studies in this area proposes a virtual heading sensor allowing each robot to be able to sense the heading direction of the other robots without requiring a goal direction. Within this sensor, the swarm could provide coordinated motion while avoiding obstacles (Turgut et al.2008). This study was extended and revealed that it is possible to insert some“informed”robots, knowing the goal direction, in the swarm so as to lead the other “non-informed” robots towards the goal direction (Ferrante et al.
2010). A novel and recent study in coordinated motionfield allows robots to change both angular and forward speed according to the computed vector without requiring an explicit alignment rule (Ferrante et al.2012).
This paper proposes a novel approach based on PFSMs and Virtual physics-based design to assign each robot into a group and allocate a leader for each group in a decentralized manner. This is different from the conventional aggrega- tion behaviour of social animals that instead of forming a single aggregate, robots are grouped according to their distance to each other and a leader is selected in a complete decentralized manner. Besides, each group relies on a centre of gravity (COG) based algorithm and navigates in a coordinated motion towards a specific goal while avoiding obstacles placed on their paths.
This paper is organized as follows. In Sect.2, the proposed framework and corresponding algorithms for multi-agent systems are presented, whereas Sect.3 focuses on implementation and evaluation of the system. The study is concluded in Sect.4.
2 Collective Behavior Framework
This section details the features of the proposed framework used for collective behaviour of multi-agent systems in a decentralized manner that comprises a generic grouping algorithms and a leader assignment procedure, followed by a coordinated navigation strategy. Flowchart of the proposed collective behaviour framework is illustrated in Fig.1. As it can be seen from the correspondingfigure, the framework consists of three main modules and two sub-modules, which will be detailed in the following sections. Essentially each robot possesses a map based navigation strategy that each robot has the 2-D map of the environment and can navigate individually towards a specific goal using a local navigation strategy.
Nevertheless, robots are not allowed to communicate each other while performing grouping, leader selection and coordinated motion behaviors. Accordingly, all these tasks are achieved in a decentralized manner.
62 M.S. Gỹzel and H. Kayakửkỹ