PRE2023 3 Group3
Group members
| Name | Student ID | Current Study Programme |
|---|---|---|
| Patryk Stefanski | 1830872 | CSE |
| Raul Sanchez Flores | 1844512 | CSE |
| Briana Isaila | 1785923 | CSE |
| Raul Hernandez Lopez | 1833197 | CSE |
| Ilie Rareş Alexandru | 1805975 | CSE |
| Ania Barbulescu | 1823612 | CSE |
Problem statement
In western society, having a family, a house and a good job are what many people aspire to have. As people strive to achieve such aspirations, their spending power increases, which allows them to be able to afford buying a nice home for their future family, with a nice garden for the kids and pets. However, as with many things, in our capitalist world, this usually comes at a sacrifice: free time. With how busy life becomes, time needed to maintain their gardens is slim, and sometimes they simply do not have the required knowledge about specific plants to appropriately take care of them.
In the past decade, robotics has been advancing across multiple fields rapidly as tedious and difficult tasks become increasingly automated, this is not any different in the field of agriculture and gardening. As the world's population continues to rapidly increase, there are increasingly many people to feed, putting pressure on farmers to optimise their crop growing and production. In recent years, many robots have become available that aid farmers in important aspects such as irrigation, plantation and weeding. These robots are large mechanical structures sold at a very high price meaning their only true usage is in large scale farming operations. Unfortunately, one common user group has been left behind and not considered when developing this new technology in gardening and agriculture, the amateur gardener. Amateur gardeners, often lacking in-depth knowledge about plants and gardening practices, face challenges in maintaining their gardens. Identifying issues with specific plants, understanding their individual needs, and implementing corrective measures can be overwhelming for their limited expertise. It is no surprise that traditional gardening tools and resources often fall short in providing the necessary guidance for optimal plant care, so another solution must be found. This is the problem that our team's robot will be the solution to.
Objectives
The objectives for the project that we hope to accomplish throughout the 8 weeks that are given to us are the following:
- The robot should be able to cut grass.
- The app which controls the robot should be user-friendly and fulfil all the robot's requirements
- The robot should be able to map out the garden's terrain and location of plants growing in it.
- The robot should be able to identify the type of a plant and if it is healthy
- The robot should be able to recommend specific actions after spotting a disease/infestation
- The app which control the robot should display the location of a unhealthy plant and recommended actions to take care of it.
Users
Who are the users?
The users of the product are garden-owners who need assistance in monitoring and maintaining their garden. This could be due to the fact that the users do not have required knowledge to properly maintain all different types of plants in their garden, or would prefer a quick and easy set of instruction of what to do with each unhealthy plant and where that plant is located. This would optimise the users routine of gardening without taking away the joy and passion that inspired the user to invest into plants in their garden in the first place.
What do the users require?
The users require a robot which is easy to operate and does not need unnecessary maintenance and setup. The robot should be easily controllable through a user interface that is tailored to the users needs and that displays all required information to the user in a clear and concise way. The user also requires that the robot may effectively map their garden and identify where certain plants are located. Lastly, the user requires that the robot is able to accurately describe what actions must be taken, if any are necessary, for a specific plant at a specific location in the garden.
Deliverables
- Research into AI plant detection mapping a garden and best ways of manoeuvring through it.
- Research into AI identifying plant diseases and infestations.
- Survey confirming that the problem we have selected to solve is a solution users desire.
- Interactive UI of an app which will allow the user to control the robot remotely that implements the user requirements that we will obtain from the survey. The UI will be able to be run on a phone and all its features will be able to be accessed through a mobile application.
- Interview with specialist in biology or AI
- This wiki page which will document the progress of the group's work, decisions that have been made and results we obtained.
State of Art
- Robots such as roombas, and automated grass-cutting robots.
- Existing plant detection AI systems
Regular Robot Operation
As with any piece of technology it is important that the users are aware of its proper operation method and how the robot functions is general. It is important that this is clear for our robot as well. Upon the robot's first use in a new garden or when the garden owner has made some changes to the garden layout, the mapping process must be instantiated in the app. This mapping will be a 2D map of the garden which will then later allow the robot to efficiently traverse the entire garden during its regular operation without leaving any part of the garden unvisited. In order to better understand this feature one, can compare it to the iRobot Roomba. After the initial setup phase has been completed the robot will be able to begin its normal operation. Normal operation includes the robot being let out into the garden from its storage place, and traversing through the garden cutting grass while its camera scans the plants in its surroundings. Whenever the robot detects an irregularity in one of the plants, it will notify the user through the user of the app, where the robot will send over a picture of the plant with an issue as well as its location on the map of the garden. The user will then able to navigate in the app to view all plants that need to be taken care of in his garden. This means that not only will the user have a lawn which is well kept but also be aware of all unhealthy plants keeping the users garden in optimal condition at all times.
.jpg)
Maneuvering: Patryk
One of the most important design decisions when creating a robot or machine with some form of mobility is deciding what mechanism the robot will use to traverse its operational environment. This decision is not always easy as many options exist which have their unique pros and cons. Therefore is is important to consider the pros and cons of all methods and then decide which method is most appropriate for a given scenario. In the following section I will explore these different methods and see which are expected to be most beneficial and work the best in the task environment our robot will be required to function in.
Wheeled Robots
It may be no surprise that the most popular method for movement within the robot industry is still a robot with circular wheels. This is due to the fact that robots with wheels are simply much easier to design and model.[1] They do not require complex mechanism of flexing or rotating a actuator but can be fully functional by simply altering rotating a motor in one of two directions. Essentially they allow the engineer to focus on the main functionality of the robot without having to worry about the many complexities that could arise with other movement mechanisms when that is not necessary. Wheeled robots are also convenient in design as they rarely take up a lot of space in the robot. Furthermore, as stated by Zedde and Yao from the University of Wagenigen, these types of robots are most often used in industry due to their simple operation and simple design.[2] Although wheeled robots seem as a single simple category there are a few subcategories of this movement mechanism that are important to distinguish as they each have their benefits and issues they face.
Differential Drive
Differential drive focuses on independent rotation of all wheels on the robot. Essentially one could say that each wheel has its own functionality and operates independently of the other wheels present on the robot. Although rotation is independent it is important to note that all wheels on the robot work as one unit to optimize turning and movement. The robot does this by varying the relative speed of rotation of its wheels which allow the robot to move in any direction without an additional steering mechanism. [3] order to better illustrate this idea consider the following scenario - suppose a robot wants to turn sharp left, the left wheels would become idle and the right wheel would rotate at maximum speed. As can be seen both wheels are rotating independently but are doing so to reach the same movement goal.
| Pros | Cons |
|---|---|
| Easy to design | Difficulty in straight line motion on uneven terrains |
| Cost-effective | Wheel skidding can completely mess up algorithm and confuse the robot of its location |
| Easy maneiveribility | Sensitive to weight distribution - big issue with moving water in container |
| Robust - less prone to mechanical failures | |
| Easy control |
Omni Directional Wheels
Omni-directional wheels are a specialized type of wheel designed with rollers or casters set at angles around their circumference. This specific configuration allows a robot which has these wheels to easily move in any direction, whether this is lateral, diagonal, or rotational motion.[4] By allowing each wheel to rotate independently and move at any angle, these wheels provide great agility and precision, which makes this method ideal for applications which require navigation and precise positioning. The main difference between this method and differential drive is the fact that omni directional wheels are able to move in any direction easily and do not require turning of the whole robot when that is not necessary due to their specially designed roller on each wheel.
| Pros | Cons |
|---|---|
| Allows complex movement patterns | Complex design and implementation |
| Superior maneuverability in any direction | Limited load-bearing capacity |
| Efficient rotation and lateral movement | Higher manufacturing costs |
| Ideal for tight spaces and precision tasks | Reduced traction on uneven terrains |
| Enhanced agility and flexibility | Susceptible to damage in rugged environments |
Legged Robots
Over millions of year organisms have evolved in thousands of different ways, giving rise to many different methods of brain functioning, how an organisms perceives the world and what is important in our current discussion, movement. It is no coincidence that many land animals have evolved to have some form of legs to traverse their habitats, it is simply a very effective method which allows a lot of versatility and adaptability to any obstacle or problem an animal might face.[5] This is no different when discussing the use of legged robots, legs provide superior functionality to many other movement mechanisms due to the fact that they are able to rotate and operate freely in all axis's. However, with great mobility comes the great cost of their very difficult design, a design with which top institutions and companies struggle with to this day.[6]
| Pros | Cons |
|---|---|
| Versatility in Terrain | Complexity in Design |
| Obstacle Negotiation | Power Consumption |
| Stability on Uneven Ground | Sensitivity to Environmental Changes |
| Human-Like Interaction | Limited Speed |
| Efficiency in Locomotion | Maintenance Challenges |
Tracked Robots
Tracked robots, which can be characterized by their continuous track systems, offer a dependable method of traversing a terrain that can be found in applications across various industries. The continuous tracks, consisting of connected links, are looped around wheels or sprockets, providing a continuous band that allows for effective and reliable movement on many different surfaces, terrains and obstacles.[7] It is therefore no surprise that their most well known usages include vehicles which operate in uneven and unpredictable, such as tanks. Since tracks are flexible it is even common that such robots can simply avoid small obstacles by driving over them without experiencing any issues. This is particularly favorable for the robot we are designing as naturally gardens are never perfectly flat surfaces often littered by many naturally cause obstacles such as stone, dents in the surface or even possibly branches that have fallen on the ground due to rough wind.
| Pros of Tracked Robots | Cons of Tracked Robots |
|---|---|
| Superior Stability | Complex Mechanical Design |
| Effective Traction | Limited Maneuverability |
| Versatility in Terrain | Terrain Alteration |
| High Payload Capacity | Increased Power Consumption |
| Efficient Over Obstacles | |
| Consistent Speed |
Hovering/Flying Robots
Hovering/Flying robots provide without a doubt the most unique way of movement from the previously listed. This method unlocks a whole new wide range of possibilities as the robot no longer has to consider on-ground obstacles; whether that is rocks or uneven terrain. This method also unlocks the possibility of the robot to optimize its movement distance as it is able to move from point A to point B directly in a straight line. However, as is the case with any solution, flying/hovering has its major problems. It is by far the most expensive method, as flying apparatus is far more costly and high maintenance than any other solution. This makes this unreliable and likely a method far out of the technological needs and requirements of our gardening robot.
| Pros | Cons |
|---|---|
| Versatile Aerial Mobility | Limited Payload Capacity |
| Rapid Deployment | Limited Endurance |
| Remote Sensing | Susceptibility to Weather |
| Reduced Ground Impact | Regulatory Restrictions |
| Dynamic Surveillance | Security and Privacy Concerns |
| Efficient Data Collection | Initial Cost and Maintenance |
Some notes from previous meeting
- We need to have an explicit link from the user needs to our product:
- ai recognition of plants. how many weeds, plants? how will the ai recognise disease? video, multiple images, 1 image?
- we need to come up with requirements for our app, and back them up with literature or the survey
- how will the robot behave in the garden? will it map out the entire garden by itself at first? how often will it use the ai to detect problems in the plants?
Work Distribution
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Define Deliverables and brainstorm new idea | Completed wiki with new idea |
| 3 | Write survey questions about garden maintenance robot, research methods to maneuver in an area, specifically in a garden. | Wiki |
| 4 | Send survey results, complete research maneuvering in gardens and begin looking into state-of-the art AI classification methods. | Wiki |
| 5 | Analyze survey results, research mapping techniques, sensors that might be required for this to be effective and complete research of AI classification models. | Wiki |
| 6 | Decide on the final requirements for the sensors and mapping of robot. | Wiki |
| 7 | Update wiki to document our progress and results | Wiki |
| 8 | Work on and finalize presentation | Final Presentation |
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Literature Review and State of the Art | Wiki |
| 3 | Write survey questions about garden maintenance robot, research about what hardware, equipment and materials
that the robot would need. |
Wiki |
| 4 | Send survey questions, complete research about hardware, equipment and materials. | Wiki |
| 5 | Analyse survey results. | Wiki |
| 6 | Decide on a final set of requirements for the hardware of the robot. | Wiki |
| 7 | update wiki to document our progress and results | Wiki |
| 8 | Work on and finalize presentation | Final Presentation. |
Research into AI identifying plant diseases and infestations: Briana.
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Research on state of the art AI. | Wiki |
| 3 | Research on plant diseases and infestations | Google Doc |
| 4 | Research on best ways to detect diseases and infestations (where to point the camera, what other sensors to use) | Google Doc |
| 5 | Research on AI state recognition (healthy/unhealthy) | Google Doc |
| 6 | Research on limitations of AI when it comes to recognising different states of a plant (healthy/unhealthy) | Google Doc |
| 7 | Conducting interviews with AI specialist. | Google Doc |
| 8 | Work on and finalize presentation. | Presentation |
Research into AI identifying plant diseases and infestations: Rareş.
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Research on state of the art AI. | Wiki |
| 3 | Research on plant diseases and infestations | Google Doc |
| 4 | Research on best ways to detect diseases and infestations (where to point the camera, what other sensors to use) | Google Doc |
| 5 | Research on AI state recognition (healthy/unhealthy) | Google Doc |
| 6 | Research on limitations of AI when it comes to recognising different states of a plant (healthy/unhealthy) | Google Doc |
| 7 | Conducting interviews with AI specialist. | Google Doc |
| 8 | Work on and finalize presentation | Presentation |
Interactive UI design and implementation: Raul H.
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Literature Review and State of the Art | Wiki |
| 3 | Write interview questions in order to find out what requirements users expect from the application, and research the Android application development process in Android Studio. | Wiki |
| 4 | Based on the interviews, compile a list of the requirements and create UI designs based on these requirements. | Wiki |
| 5 | Start implementing the UI designs into a functional application in Android Studio. | |
| 6 | Finish implementing the UI designs into a functional application in Android Studio. | Completed demo application |
| 7 | ||
| 8 | Work on and finalize presentation |
Interactive UI design and implementation: Ania
| Week | Task | Deliverable to meeting |
|---|---|---|
| 2 | Literature Review and State of the Art of Garden Robots and Plant Recognition Software | Wiki |
| 3 | Write interview questions in order to find out what requirements users expect from the application, start creating UI design based on current concept ideas. | Wiki |
| 4 | Based on the interviews, compile a list of the requirements and create UI designs based on these requirements. | Wiki |
| 5 | Start implementing the UI designs into a functional application in Android Studio. | |
| 6 | Finish implementing the UI designs into a functional application in Android Studio. | Completed demo application |
| 7 | Testing and final changes to UI design. | |
| 8 | Work on and finalize presentation |
Individual effort
| Break-down of hours | Total Hours Spent | ||
|---|---|---|---|
| Week 1 | Patryk Stefanski | Attended kick-off (2h), Research into subject idea (2h), Meet with group to discuss ideas (2h), Reading Literature (2h), Updating wiki (1h) | 9 |
| Raul Sanchez Flores | Attended kick-off (2h) | 2 | |
| Briana Isaila | Attended kick-off (2h), Meet with group to discuss ideas (2h), Research state of the art (2h), Look into different ideas (2h) | 8 | |
| Raul Hernandez Lopez | |||
| Ilie Rareş Alexandru | |||
| Ania Barbulescu | Attended kick-off (2h), Reading Literature (2h) | 4 | |
| Week 2 | Patryk Stefanski | Meeting with tutors (0.5h), Researched and found contact person who maintains Dommel (1h), Brainstorming new project ideas (3h), Group meeting Thursday (1.5h), Created list of possible deliverables (1h), Group meeting to establish tasks (4.5h), Literature review and updated various parts of wiki (2h) | 13.5 |
| Raul Sanchez Flores | Meeting with tutors (0.5h), Group meeting Thursday (1.5h) | 2 | |
| Briana Isaila | Meeting with tutors (0.5h), Group meeting Thursday (1.5h), Brainstorming new project ideas (2h), Updating wiki (2h), Group meeting to establish tasks (4.5h) | 10.5 | |
| Raul Hernandez Lopez | Meeting with tutors (0.5h), Group meeting Thursday (1.5h) | 2 | |
| Ilie Rareş Alexandru | Meeting with tutors (0.5h), Group meeting Thursday (1.5h) | 2 | |
| Ania Barbulescu | Group meeting Friday (4h), Research Literature (2h), Updated Wiki (2h) | 8 | |
| Week 3 | Patryk Stefanski | Meeting with tutors (0.5h), Research to specify problem more concretely (3.5h), Found literature that backs up problem is necessary (1h), Group meeting Tuesday (1h), Finished Problem statement, objectives, users (2h), Research into maneuvering and reporting on findings (3h) | 11 |
| Raul Sanchez Flores | |||
| Briana Isaila | Meeting with tutors (0.5h) | ||
| Raul Hernandez Lopez | |||
| Ilie Rareş Alexandru | |||
| Ania Barbulescu |
Literature Review
1. TrimBot2020: an outdoor robot for automatic gardening (https://www.researchgate.net/publication/324245899_TrimBot2020_an_outdoor_robot_for_automatic_gardening)
· The TrimBot2020 program aims to build a prototype of the world’s first outdoor robot for automatic bush trimming and rose pruning.
· State of the art: ‘green thumb’ robots used for automatic planting and harvesting.
· Gardens pose a variety of hurdles for autonomous systems by virtue of being dynamic environments: natural growth of plants and flowers, variable lighting conditions, as well as varying weather conditions all influence the appearance of objects in the environment.
· Additionally, the terrain is often uneven and contains areas that are difficult for a robot to navigate, such as those made of pebbles or woodchips.
· The design of the TrimBot2020 is based on the Bosch Indigo lawn mower, on which a Kinova robotic arm is then mounted. (It might, therefore, be worthwhile to research both of these technologies.)
· The robot’s vision system consists of five pairs of stereo cameras arranged such that they offer a 360◦ view of the environment. Additionally, each stereo pair is comprised of one RGB camera and one grayscale camera.
· The robot uses a Simultaneous Localization and Mapping (SLAM) system in order to move through the garden. The system is responsible for simultaneously estimating a 3D map of the garden in the form of a sparse point cloud and the position of the robot in respect to the resulting 3D map.
· For understanding the environment and operating the robotic arm, TrimBot2020 has developed algorithms for disparity computation from monocular images and from stereo images, based on convolutional neural networks, 3D plane labeling and trinocular matching with baseline recovery. An algorithm for optical flow estimation was also developed, based on a multi-stage CNN approach with iterative refinement of its own predictions.
2. Robots in the Garden: Artificial Intelligence and Adaptive Landscapes (https://www.researchgate.net/publication/370949019_Robots_in_the_Garden_Artificial_Intelligence_and_Adaptive_Landscapes)
· FarmBot is a California-based firm and designs and markets open-source commercial gardening robots, and develops web applications for users to interface with these robots.
· These robots employ interchangeable tool heads to rake soil, plant seeds, water plants, and weed. They are highly customizable: users can design and replace most parts to suit their individual needs. In addition to that, FarmBot’s code is open source, allowing users to customize it through an online web app.
· Initially, the user describes the garden’s contents to a FarmBot as a simple placement of plants from the provided plant dictionary on a garden map, a two-dimensional grid visualized by the web app. FarmBot stores the location of each plant as a datapoint (x, y) on that map. Other emerging plants, if detected by the camera, are treated uniformly as weeds that should be managed by the robot.
· The robotic vision system employed by the Ecological Laboratory for Urban Agriculture consists of AI cameras that process images with OpenCV, an open-source computer vision and machine learning software library. This library provides machine learning algorithms, including pre-trained deep neural network modules that can be modified and used for specific tasks, such as measuring plant canopy coverage and plant height.
3. Indoor Robot Gardening: Design and Implementation (https://www.researchgate.net/publication/225485587_Indoor_robot_gardening_Design_and_implementation)
·
4. Building a Distributed Robot Garden (https://www.researchgate.net/publication/224090704_Building_a_Distributed_Robot_Garden)
5. A robotic irrigation system for urban gardening and agriculture (https://www.researchgate.net/publication/337580011_A_robotic_irrigation_system_for_urban_gardening_and_agriculture)
6. Design and Implementation of an Urban Farming Robot (https://www.researchgate.net/publication/358882608_Design_and_Implementation_of_an_Urban_Farming_Robot)
7. Small Gardening Robot with Decision-making Watering System (https://www.researchgate.net/publication/363730362_Small_gardening_robot_with_decision-making_watering_system)
8. A cognitive architecture for automatic gardening (https://www.researchgate.net/publication/316594452_A_cognitive_architecture_for_automatic_gardening)
9. Recent Advancements in Agriculture Robots: Benefits and Challenges (https://www.researchgate.net/publication/366795395_Recent_Advancements_in_Agriculture_Robots_Benefits_and_Challenges)
10. A Survey of Robot Lawn Mowers (https://www.researchgate.net/publication/235679799_A_Survey_of_Robot_Lawn_Mowers)
11. Distributed Gardening System Using Object Recognition and Visual Servoing (https://www.researchgate.net/publication/341788340_Distributed_Gardening_System_Using_Object_Recognition_and_Visual_Servoing)
12. A Plant Recognition Approach Using Shape and Color Features in Leaf Images (https://www.researchgate.net/publication/278716340_A_Plant_Recognition_Approach_Using_Shape_and_Color_Features_in_Leaf_Images)
13. A study on plant recognition using conventional image processing and deep learning approaches (https://www.researchgate.net/publication/330492923_A_study_on_plant_recognition_using_conventional_image_processing_and_deep_learning_approaches)
14. Plant Recognition from Leaf Image through Artificial Neural Network (https://www.researchgate.net/publication/258789208_Plant_Recognition_from_Leaf_Image_through_Artificial_Neural_Network)
15. Deep Learning for Plant Identification in Natural Environment (https://www.researchgate.net/publication/317127150_Deep_Learning_for_Plant_Identification_in_Natural_Environment)
16. Identification of Plant Species by Deep Learning and Providing as A Mobile Application (https://www.researchgate.net/publication/348008139_Identification_of_Plant_Species_by_Deep_Learning_and_Providing_as_A_Mobile_Application)
17. Path Finding Algorithms for Navigation (https://nl.mathworks.com/help/nav/ug/choose-path-planning-algorithms-for-navigation.html)
18. NetLogo Models (https://ccl.northwestern.edu/netlogo/models/)
19. Path Finding Algorithms (https://neo4j.com/developer/graph-data-science/path-finding-graph-algorithms/#:~:text=Path%20finding%20algorithms%20build%20on,number%20of%20hops%20or%20weight.)
20. How Navigation Agents Learn About Their Environment (https://openaccess.thecvf.com/content/CVPR2022/papers/Dwivedi_What_Do_Navigation_Agents_Learn_About_Their_Environment_CVPR_2022_paper.pdf)
21. NetLogo Library (https://ccl.northwestern.edu/netlogo/docs/dictionary.html
22. Adapting to a Robot: Adapting Gardening and the Garden to fit a Robot Lawn Mower (https://dl.acm.org/doi/abs/10.1145/3371382.3380738)
23. Automatic Distributed Gardening System Using Object Recognition and Visual Servoing (https://link.springer.com/chapter/10.1007/978-981-15-7345-3_30)
24. An Overview Of Smart Garden Automation (https://ieeexplore.ieee.org/abstract/document/9077615)
25. A cognitive architecture for automatic gardening (https://www.sciencedirect.com/science/article/pii/S0168169916304768)