PRE2023 3 Group3

Group members

Name	Student ID	Current Study Programme
Patryk Stefanski	1830872	CSE
Raul Sanchez Flores	1844512	CSE
Briana Isaila	1785923	CSE
Raul Hernandez Lopez	1833197	CSE
Ilie Rareş Alexandru	1805975	CSE
Ania Barbulescu	1823612	CSE

Problem statement

In western society, having a family, a house and a good job are what many people aspire to have. As people strive to achieve such aspirations, their spending power increases, which allows them to be able to afford buying a nice home for their future family, with a nice garden for the kids and pets. However, as with many things, in our capitalist world, this usually comes at a sacrifice: free time. With how busy life becomes, time needed to maintain their gardens is slim, and sometimes they simply do not have the required knowledge about specific plants to appropriately take care of them.

In the past decade, robotics has been advancing across multiple fields rapidly as tedious and difficult tasks become increasingly automated, this is not any different in the field of agriculture and gardening. As the world's population continues to rapidly increase, there are increasingly many people to feed, putting pressure on farmers to optimise their crop growing and production. In recent years, many robots have become available that aid farmers in important aspects such as irrigation, plantation and weeding. These robots are large mechanical structures sold at a very high price meaning their only true usage is in large scale farming operations. Unfortunately, one common user group has been left behind and not considered when developing this new technology in gardening and agriculture, the amateur gardener. Amateur gardeners, often lacking in-depth knowledge about plants and gardening practices, face challenges in maintaining their gardens. Identifying issues with specific plants, understanding their individual needs, and implementing corrective measures can be overwhelming for their limited expertise. It is no surprise that traditional gardening tools and resources often fall short in providing the necessary guidance for optimal plant care, so another solution must be found. This is the problem that our team's robot will be the solution to.

Objectives

The objectives for the project that we hope to accomplish throughout the 8 weeks that are given to us are the following:

• The robot should be able to cut grass.
• The app which controls the robot should be user-friendly and fulfil all the robot's requirements
• The robot should be able to map out the garden's terrain and location of plants growing in it.
• The robot should be able to identify the type of a plant and if it is healthy
• The robot should be able to recommend specific actions after spotting a disease/infestation
• The app which control the robot should display the location of a unhealthy plant and recommended actions to take care of it.
  

Users

Who are the users?

The users of the product are garden-owners who need assistance in monitoring and maintaining their garden. This could be due to the fact that the users do not have required knowledge to properly maintain all different types of plants in their garden, or would prefer a quick and easy set of instruction of what to do with each unhealthy plant and where that plant is located. This would optimise the users routine of gardening without taking away the joy and passion that inspired the user to invest into plants in their garden in the first place.

What do the users require?

The users require a robot which is easy to operate and does not need unnecessary maintenance and setup. The robot should be easily controllable through a user interface that is tailored to the users needs and that displays all required information to the user in a clear and concise way. The user also requires that the robot may effectively map their garden and identify where certain plants are located. Lastly, the user requires that the robot is able to accurately describe what actions must be taken, if any are necessary, for a specific plant at a specific location in the garden.

Deliverables

• Research into AI plant detection mapping a garden and best ways of manoeuvring through it.
• Research into AI identifying plant diseases and infestations.
• Survey confirming that the problem we have selected to solve is a solution users desire.
• Interactive UI of an app which will allow the user to control the robot remotely that implements the user requirements that we will obtain from the survey. The UI will be able to be run on a phone and all its features will be able to be accessed through a mobile application.
• Interview with specialist in biology or AI
• This wiki page which will document the progress of the group's work, decisions that have been made and results we obtained.

State of Art

• Robots such as roombas, and automated grass-cutting robots.
• Existing plant detection AI systems

Work Distribution

Navigation Algorithm and Netlogo Implementation: Patryk

Week	Task	Deliverable to meeting
2	Define Deliverables and brainstorm new idea	Completed wiki with new idea
3	Write survey questions about garden maintenance robot, research ways maneuver in an area, specifically in a garden.	Wiki
4	Send survey results, complete research maneuvering in gardens and begin looking into state-of-the art AI classification methods.	Wiki
5	Analyze survey results, research mapping techniques, sensors that might be required for this to be effective and complete research of AI classification models.	Wiki
6	Decide on the final requirements for the sensors and mapping of robot.	Wiki
7	Update wiki to document our progress and results	Wiki
8	Work on and finalize presentation	Final Presentation

Navigation Algorithm and Netlogo Implementation: Raul S.

Week	Task	Deliverable to meeting
2	Literature Review and State of the Art	Wiki
3	Write survey questions about garden maintenance robot, research about what hardware, equipment and materials that the robot would need.	Wiki
4	Send survey questions, complete research about hardware, equipment and materials.	Wiki
5	Analyse survey results.	Wiki
6	Decide on a final set of requirements for the hardware of the robot.	Wiki
7	update wiki to document our progress and results	Wiki
8	Work on and finalize presentation	Final Presentation.

Research into AI identifying plant diseases and infestations: Briana.

Week	Task	Deliverable to meeting
2	Research on state of the art.	Wiki
3	Research on plant diseases	Wiki
4	Research on plant infestations	Google Doc
5		Google Doc
6		Google Doc
7
8	Work on and finalize presentation.	Presentation

Research into AI identifying plant diseases and infestations: Rareş.

Week	Task	Deliverable to meeting
2	Research on state of the art.	Wiki
3		Wiki
4		Google Doc
5		Google Doc
6		Google Doc
7		CAD
8	Work on and finalize presentation	Presentation

Interactive UI design and implementation: Raul H.

Week	Task	Deliverable to meeting
2	Literature Review and State of the Art	Wiki
3	Write interview questions in order to find out what requirements users expect from the application, and research the Android application development process in Android Studio.	Wiki
4	Based on the interviews, compile a list of the requirements and create UI designs based on these requirements.	Wiki
5	Start implementing the UI designs into a functional application in Android Studio.
6	Finish implementing the UI designs into a functional application in Android Studio.	Completed demo application
7
8	Work on and finalize presentation

Interactive UI design and implementation: Ania

Week	Task	Deliverable to meeting
2	Literature Review and State of the Art of Garden Robots and Plant Recognition Software	Wiki
3	Write interview questions in order to find out what requirements users expect from the application, start creating UI design based on current concept ideas.	Wiki
4	Based on the interviews, compile a list of the requirements and create UI designs based on these requirements.	Wiki
5	Start implementing the UI designs into a functional application in Android Studio.
6	Finish implementing the UI designs into a functional application in Android Studio.	Completed demo application
7	Testing and final changes to UI design.
8	Work on and finalize presentation

Individual effort

		Break-down of hours	Total Hours Spent
Week 1	Patryk Stefanski	Attended kick-off (2h), Research into subject idea (2h), Meet with group to discuss ideas (2h), Reading Literature (2h), Updating wiki (1h)	9
	Raul Sanchez Flores	Attended kick-off (2h)	2
	Briana Isaila	Attended kick-off (2h), Meet with group to discuss ideas (2h), Research state of the art (2h), Look into different ideas (2h)	8
	Raul Hernandez Lopez
	Ilie Rareş Alexandru
	Ania Barbulescu	Attended kick-off (2h), Reading Literature (2h)	4
Week 2	Patryk Stefanski	Meeting with tutors (0.5h), Researched and found contact person who maintains Dommel (1h), Brainstorming new project ideas (2h), Group meeting Thursday (1.5h), Created list of possible deliverables (1h), Group meeting to establish tasks (4.5h), Literature review and updated various parts of wiki (2h)	12.5
	Raul Sanchez Flores	Meeting with tutors (0.5h), Group meeting Thursday (1.5h)	2
	Briana Isaila	Meeting with tutors (0.5h), Group meeting Thursday (1.5h), Brainstorming new project ideas (2h), Updating wiki (2h), Group meeting to establish tasks (4.5h)	10.5
	Raul Hernandez Lopez	Meeting with tutors (0.5h), Group meeting Thursday (1.5h)	2
	Ilie Rareş Alexandru	Meeting with tutors (0.5h), Group meeting Thursday (1.5h)	2
	Ania Barbulescu	Group meeting Friday (4h), Research Literature (2h), Updated Wiki (2h)	8
Week 3	Patryk Stefanski	Meeting with tutors (0.5h), Research to specify problem more concretely (3.5h), Found literature that backs up problem is necessary (1h), Group meeting Tuesday (1h), Finished Problem statement, objectives, users (2h)	8
	Raul Sanchez Flores
	Briana Isaila	Meeting with tutors (0.5h)
	Raul Hernandez Lopez
	Ilie Rareş Alexandru
	Ania Barbulescu

Literature Review

1.     TrimBot2020: an outdoor robot for automatic gardening (https://www.researchgate.net/publication/324245899_TrimBot2020_an_outdoor_robot_for_automatic_gardening)

·      The TrimBot2020 program aims to build a prototype of the world’s first outdoor robot for automatic bush trimming and rose pruning.

·      State of the art: ‘green thumb’ robots used for automatic planting and harvesting.

·      Gardens pose a variety of hurdles for autonomous systems by virtue of being dynamic environments: natural growth of plants and flowers, variable lighting conditions, as well as varying weather conditions all influence the appearance of objects in the environment.

·      Additionally, the terrain is often uneven and contains areas that are difficult for a robot to navigate, such as those made of pebbles or woodchips.

·      The design of the TrimBot2020 is based on the Bosch Indigo lawn mower, on which a Kinova robotic arm is then mounted. (It might, therefore, be worthwhile to research both of these technologies.)

·      The robot’s vision system consists of five pairs of stereo cameras arranged such that they offer a 360◦ view of the environment. Additionally, each stereo pair is comprised of one RGB camera and one grayscale camera.

·      The robot uses a Simultaneous Localization and Mapping (SLAM) system in order to move through the garden. The system is responsible for simultaneously estimating a 3D map of the garden in the form of a sparse point cloud and the position of the robot in respect to the resulting 3D map.

·      For understanding the environment and operating the robotic arm, TrimBot2020 has developed algorithms for disparity computation from monocular images and from stereo images, based on convolutional neural networks, 3D plane labeling and trinocular matching with baseline recovery. An algorithm for optical flow estimation was also developed, based on a multi-stage CNN approach with iterative refinement of its own predictions.

2.     Robots in the Garden: Artificial Intelligence and Adaptive Landscapes (https://www.researchgate.net/publication/370949019_Robots_in_the_Garden_Artificial_Intelligence_and_Adaptive_Landscapes)

·      FarmBot is a California-based firm and designs and markets open-source commercial gardening robots, and develops web applications for users to interface with these robots.

·      These robots employ interchangeable tool heads to rake soil, plant seeds, water plants, and weed. They are highly customizable: users can design and replace most parts to suit their individual needs. In addition to that, FarmBot’s code is open source, allowing users to customize it through an online web app.

·      Initially, the user describes the garden’s contents to a FarmBot as a simple placement of plants from the provided plant dictionary on a garden map, a two-dimensional grid visualized by the web app. FarmBot stores the location of each plant as a datapoint (x, y) on that map. Other emerging plants, if detected by the camera, are treated uniformly as weeds that should be managed by the robot.

·      The robotic vision system employed by the Ecological Laboratory for Urban Agriculture consists of AI cameras that process images with OpenCV, an open-source computer vision and machine learning software library. This library provides machine learning algorithms, including pre-trained deep neural network modules that can be modified and used for specific tasks, such as measuring plant canopy coverage and plant height.

3.      Indoor Robot Gardening: Design and Implementation (https://www.researchgate.net/publication/225485587_Indoor_robot_gardening_Design_and_implementation)

·      

4. Building a Distributed Robot Garden (https://www.researchgate.net/publication/224090704_Building_a_Distributed_Robot_Garden)

5. A robotic irrigation system for urban gardening and agriculture (https://www.researchgate.net/publication/337580011_A_robotic_irrigation_system_for_urban_gardening_and_agriculture)

6. Design and Implementation of an Urban Farming Robot (https://www.researchgate.net/publication/358882608_Design_and_Implementation_of_an_Urban_Farming_Robot)

7. Small Gardening Robot with Decision-making Watering System (https://www.researchgate.net/publication/363730362_Small_gardening_robot_with_decision-making_watering_system)

8. A cognitive architecture for automatic gardening (https://www.researchgate.net/publication/316594452_A_cognitive_architecture_for_automatic_gardening)

9. Recent Advancements in Agriculture Robots: Benefits and Challenges (https://www.researchgate.net/publication/366795395_Recent_Advancements_in_Agriculture_Robots_Benefits_and_Challenges)

10. A Survey of Robot Lawn Mowers (https://www.researchgate.net/publication/235679799_A_Survey_of_Robot_Lawn_Mowers)

11. Distributed Gardening System Using Object Recognition and Visual Servoing (https://www.researchgate.net/publication/341788340_Distributed_Gardening_System_Using_Object_Recognition_and_Visual_Servoing)

12. A Plant Recognition Approach Using Shape and Color Features in Leaf Images (https://www.researchgate.net/publication/278716340_A_Plant_Recognition_Approach_Using_Shape_and_Color_Features_in_Leaf_Images)

13. A study on plant recognition using conventional image processing and deep learning approaches (https://www.researchgate.net/publication/330492923_A_study_on_plant_recognition_using_conventional_image_processing_and_deep_learning_approaches)

14. Plant Recognition from Leaf Image through Artificial Neural Network (https://www.researchgate.net/publication/258789208_Plant_Recognition_from_Leaf_Image_through_Artificial_Neural_Network)

15. Deep Learning for Plant Identification in Natural Environment (https://www.researchgate.net/publication/317127150_Deep_Learning_for_Plant_Identification_in_Natural_Environment)

16. Identification of Plant Species by Deep Learning and Providing as A Mobile Application (https://www.researchgate.net/publication/348008139_Identification_of_Plant_Species_by_Deep_Learning_and_Providing_as_A_Mobile_Application)

17. Path Finding Algorithms for Navigation (https://nl.mathworks.com/help/nav/ug/choose-path-planning-algorithms-for-navigation.html)

18. NetLogo Models (https://ccl.northwestern.edu/netlogo/models/)

19. Path Finding Algorithms (https://neo4j.com/developer/graph-data-science/path-finding-graph-algorithms/#:~:text=Path%20finding%20algorithms%20build%20on,number%20of%20hops%20or%20weight.)

20. How Navigation Agents Learn About Their Environment (https://openaccess.thecvf.com/content/CVPR2022/papers/Dwivedi_What_Do_Navigation_Agents_Learn_About_Their_Environment_CVPR_2022_paper.pdf)

21. NetLogo Library (https://ccl.northwestern.edu/netlogo/docs/dictionary.html

22. Adapting to a Robot: Adapting Gardening and the Garden to fit a Robot Lawn Mower (https://dl.acm.org/doi/abs/10.1145/3371382.3380738)

23. Automatic Distributed Gardening System Using Object Recognition and Visual Servoing (https://link.springer.com/chapter/10.1007/978-981-15-7345-3_30)

24. An Overview Of Smart Garden Automation (https://ieeexplore.ieee.org/abstract/document/9077615)

25. A cognitive architecture for automatic gardening (https://www.sciencedirect.com/science/article/pii/S0168169916304768)