PRE2022 3 Group4: Difference between revisions

From Control Systems Technology Group
Jump to navigation Jump to search
 
(45 intermediate revisions by 6 users not shown)
Line 1: Line 1:
<div style="font-family: 'Arial'; font-size: 16px; line-height: 1.5; max-width: 1100px; word-wrap: break-word; color: #333; font-weight: 400; box-shadow: 0px 25px 35px -5px rgba(0,0,0,0.75); margin-left: auto; margin-right: auto; padding: 70px; background-color: white; padding-top: 25px;">
----
''As written by ChatGPT:''<blockquote>Artificial Intelligence, a wonder of the modern age  
''As written by ChatGPT:''<blockquote>Artificial Intelligence, a wonder of the modern age  


Line 45: Line 49:
|Psychology & Technology
|Psychology & Technology
|}
|}
<br />


==Introduction==
==Introduction==
The history of ChatGPT can be traced back to 2018, when OpenAI, an artificial intelligence research organization, released the first version of their language model called GPT-1 (Generative Pre-trained Transformer). GPT-1 was trained on a massive corpus of text data and was capable of generating human-like text. This initial release sparked interest and excitement in the natural language processing community. In June 2020, OpenAI released an upgraded version of the model called GPT-3, which had 175 billion parameters, making it the largest language model in existence at the time. GPT-3 quickly became popular due to its impressive ability to generate coherent and human-like responses in a wide range of contexts. However, due to the computational power required to run it, GPT-3 was not accessible to everyone. In 2021, OpenAI released an updated version of GPT-3 called GPT-3.5, which was specifically designed for deployment on a wider range of hardware, including mobile devices. GPT-3.5 included many of the same features as GPT-3, including its impressive natural language generation capabilities and its ability to perform various language tasks such as translation, summarization, and question answering. However, it was optimized for better performance on smaller devices.<ref name=":2" />
ChatGPT can be tracked back to 2018, to when a first version called Generative Pre-trained Transformer, or GPT-1 for short, was released. This was a very exciting development, however it was not very advanced yet, and therefore did not become very popular. In June 2020, OpenAI released an upgraded version of the GPT-1 model, named GPT-3. This model had a lot more parameters, making it the largest and most extensive language model at that time. It became very popular very quickly, due to its power. This model could generate coherent and human-like responses to a lot of questions, something that the world had not seen before. However, it did take quite some computational power to run it, and therefore GPT-3 was not available for everyone to use. In 2021 OpenAI released yet another updated version, called GPT-3.5, specially designed to be more accessible, and be able to run on a wider range of hardware, even including mobile phones.<ref name=":2" /> On November 30th of 2022, the newest version was released, called ChatGPT. This model was built upon the GPT-3.5 version, and was fine-tuned. It has taken the world by storm by what it can do.


One of the most significant applications of GPT-3 and ChatGPT has been in the development of chatbots and virtual assistants. These systems use the natural language processing capabilities of the models to simulate human-like conversations and provide assistance and support to users. ChatGPT has also been used for other applications, such as content generation, language translation, and text completion. The development of ChatGPT and other large language models has opened up new possibilities for natural language processing and artificial intelligence. It has also raised ethical concerns, such as the potential for the misuse of the technology and the impact on the job market. However, the potential benefits of ChatGPT and other language models in various fields, including education, healthcare, and communication, cannot be ignored. It will be exciting to see how these models continue to evolve and impact our lives in the years to come.<ref name=":3" />
The function of ChatGPT that it is used the most, is as a chatbot or virtual assistant. Because of the natural language processing capabilities of ChatGPT, it can simulate human-like conversations rather well and provide its users with support and assistance. ChatGPT has other important uses as well, as it can be used for content generation, language translation and text completion. It is a really exciting new technology, as it has opened up possibilities for natural language processing and artificial intelligence (AI). ChatGPT can do a lot, which also raises ethical concerns, such as in general the potential for misuse of the technology, and how it will impact the job market. Potential concerns aside, ChatGPT also brings us benefits, like its use in education, healthcare and communication. It will be exciting to see how ChatGPT and other models will continue to evolve and impact our lives in the upcoming years.<ref name=":3" />


ChatGPT is a remarkable technological advancement that has revolutionized natural language processing, and its impact on the field of education has been significant. With its ability to generate human-like responses and hold coherent conversations on a wide range of topics, ChatGPT has proven to be an essential tool for education in various ways. Certainly at the times we are living in now, just coming out of an global pandemic, ChatGPT has the power to revolutionize the academic field<ref name=":6">Chaudhry, M.A., Kazim, E. Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021. ''AI Ethics'' 2, 157–165 (2022). <nowiki>https://doi.org/10.1007/s43681-021-00074-z</nowiki></ref>. At the moment there are major problems with education, such as lack of teaching practice experience, outmoded curriculums, lack of independent learning and lack of innovation in teaching methods<ref name=":7">Shuai Yang & Haicheng Bai (2020). The integration design of artificial intelligence and normal students' Education. ''Journal of Physics: Conference Series'', Volume 1453: Conf. Ser. 1453 012090. https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012090</ref>. These problems can be combated with implementing AI such as ChatGPT in education.  
When focusing more on the field of academic education, ChatGPT has been revolutionary and has had a huge impact. Because of its ability to generate human-like text responses for almost every topic imaginable, it has been proven to be an essential tool for education. Certainly for the times we are living in now, just coming out of a global pandemic, ChatGPT has the power to revolutionize the academic field<ref name=":6">Chaudhry, M.A., Kazim, E. Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021. ''AI Ethics'' 2, 157–165 (2022). <nowiki>https://doi.org/10.1007/s43681-021-00074-z</nowiki></ref>. At the moment there are major problems with education, such as lack of practical teaching experience, outmoded curriculums, lack of independent learning and lack of innovation in teaching methods<ref name=":7">Shuai Yang & Haicheng Bai (2020). The integration design of artificial intelligence and normal students' Education. ''Journal of Physics: Conference Series'', Volume 1453: Conf. Ser. 1453 012090. https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012090</ref>. These problems can be combated with implementing AI such as ChatGPT in education.  


One of the significant ways that ChatGPT has influenced education is by providing a personalized learning experience<ref name=":8">Qadir, J. (2022). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education.</ref><ref name=":9">Limna, Pongsakorn and Jakwatanatham, Somporch and Siripipattanakul, Sutithep and Kaewpuang, Pichart and Sriboonruang, Patcharavadee, A Review of Artificial Intelligence (AI) in Education during the Digital Era (July 2022). Advance Knowledge for Executives, 1(1), No. 3, 1-9, 2022, Available at SSRN: <nowiki>https://ssrn.com/abstract=4160798</nowiki></ref>. The model has the capacity to analyze student data and offer tailored learning materials based on the individual's needs, interests, and preferences. This feature is especially important in today's educational landscape, where traditional one-size-fits-all teaching methods are becoming increasingly outdated. By providing personalized learning experiences, ChatGPT has the potential to improve learning outcomes and increase student engagement and motivation. In addition, AI can also help streamline administrative tasks, such as grading and record-keeping, freeing up teachers to focus on instruction and interaction with students.<ref name=":4" /><ref name=":11">Beth McMurtrie (2018, August 12). How Artificial Intelligence Is Changing Teaching. ''The Chronicle of Higher Education''(1). Retrieved from https://www.su.edu/conservatory/files/2018/09/How-Artificial-Intelligence-is-Changing-Teaching.pdf</ref> Teachers will become more efficient which could in result increase the quality of education.<ref name=":12">Alam, A. (2021b). Should Robots Replace Teachers? Mobilisation of AI and Learning Analytics in Education. ''2021 International Conference on Advances in Computing, Communication, and Control (ICAC3)''. <nowiki>https://doi.org/10.1109/icac353642.2021.9697300</nowiki></ref><ref name=":1">Ahmad SF, Alam MM, Rahmat MK, Mubarik MS, Hyder SI. Academic and Administrative Role of Artificial Intelligence in Education. ''Sustainability''. 2022; 14(3):1101. <nowiki>https://doi.org/10.3390/su14031101</nowiki></ref> It is however important that teachers get trained in ChatGPT and feel comfortable with its use, because if the teacher is uncomfortable with the AI, the student is less likely to adopt it.<ref name=":13">Kim, J., Merrill, K., Xu, K., & Sellnow, D. D. (2020). My Teacher Is a Machine: Understanding Students’ Perceptions of AI Teaching Assistants in Online Education. ''International Journal of Human–Computer Interaction'', ''36''(20), 1902–1911. <nowiki>https://doi.org/10.1080/10447318.2020.1801227</nowiki></ref> So it is important to help teachers to develop necessary digital competencies and skills for using ChatGPT in ethical and informed ways to enhance the student learning experience and attainment of learning outcomes.<ref name=":14">Lameras, P., & Arnab, S. (2021). Power to the Teachers: An Exploratory Review on Artificial Intelligence in Education. ''Information'', ''13''(1), 14. MDPI AG. Retrieved from http://dx.doi.org/10.3390/info13010014</ref>
One of the significant ways that ChatGPT has influenced education is by providing a personalized learning experience<ref name=":8">Qadir, J. (2022). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education.</ref><ref name=":9">Limna, Pongsakorn and Jakwatanatham, Somporch and Siripipattanakul, Sutithep and Kaewpuang, Pichart and Sriboonruang, Patcharavadee, A Review of Artificial Intelligence (AI) in Education during the Digital Era (July 2022). Advance Knowledge for Executives, 1(1), No. 3, 1-9, 2022, Available at SSRN: <nowiki>https://ssrn.com/abstract=4160798</nowiki></ref>. The model has the capacity to analyse student data and offer tailored learning materials based on the individual's needs, interests, and preferences. This feature is especially important in today's educational landscape, where traditional one-size-fits-all teaching methods are becoming increasingly outdated. By providing personalized learning experiences, ChatGPT has the potential to improve learning outcomes and increase student engagement and motivation. In addition, AI can also help streamline administrative tasks, such as grading and record-keeping, freeing up teachers to focus on instruction and interaction with students.<ref name=":4" /><ref name=":11">Beth McMurtrie (2018, August 12). How Artificial Intelligence Is Changing Teaching. ''The Chronicle of Higher Education''(1). Retrieved from https://www.su.edu/conservatory/files/2018/09/How-Artificial-Intelligence-is-Changing-Teaching.pdf</ref> Teachers will become more efficient, which could in result increase the quality of education.<ref name=":12">Alam, A. (2021b). Should Robots Replace Teachers? Mobilisation of AI and Learning Analytics in Education. ''2021 International Conference on Advances in Computing, Communication, and Control (ICAC3)''. <nowiki>https://doi.org/10.1109/icac353642.2021.9697300</nowiki></ref><ref name=":1">Ahmad SF, Alam MM, Rahmat MK, Mubarik MS, Hyder SI. Academic and Administrative Role of Artificial Intelligence in Education. ''Sustainability''. 2022; 14(3):1101. <nowiki>https://doi.org/10.3390/su14031101</nowiki></ref> It is however important that teachers get trained in ChatGPT and feel comfortable with using it, for if the teacher is uncomfortable with the AI, the students are less likely to adopt it.<ref name=":13">Kim, J., Merrill, K., Xu, K., & Sellnow, D. D. (2020). My Teacher Is a Machine: Understanding Students’ Perceptions of AI Teaching Assistants in Online Education. ''International Journal of Human–Computer Interaction'', ''36''(20), 1902–1911. <nowiki>https://doi.org/10.1080/10447318.2020.1801227</nowiki></ref> So it is important to help teachers to develop necessary digital competencies and skills for using ChatGPT in ethical and informed ways to enhance the student learning experience and attainment of learning outcomes.<ref name=":14">Lameras, P., & Arnab, S. (2021). Power to the Teachers: An Exploratory Review on Artificial Intelligence in Education. ''Information'', ''13''(1), 14. MDPI AG. Retrieved from http://dx.doi.org/10.3390/info13010014</ref>


Another way that ChatGPT has impacted education is through its language translation capabilities. With ChatGPT, language barriers are no longer a significant obstacle to learning. The model can translate text and speech in real-time, making it possible for students and educators to communicate effectively across different languages and cultures. This feature has opened up new possibilities for global collaboration and knowledge-sharing, and it has also made education more accessible to students from diverse backgrounds. Moreover, ChatGPT has become an essential tool for chatbots and virtual assistants used in education. Chatbots and virtual assistants have become increasingly popular in educational settings as they can provide 24/7 support to students and answer their questions promptly. For example, new developments in AI can solve 81% of university-level mathematics questions and even come up with new questions.<ref name=":15">Drori, I., Zhang, S. X., Shuttleworth, R., Tang, L., Lu, A., Elizabeth, K. E., Liu, K. X., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N. K., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G. (2022). A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. ''Proceedings of the National Academy of Sciences of the United States of America'', ''119''(32). <nowiki>https://doi.org/10.1073/pnas.2123433119</nowiki></ref> With ChatGPT's ability to hold coherent conversations, these assistants can simulate human interactions and provide personalized support to students, thereby enhancing their learning experience.<ref name=":5" />
Another way that ChatGPT has impacted education is through its language translation capabilities. With ChatGPT, language barriers are no longer a significant obstacle to learning. The model can translate text and speech in real-time, making it possible for students and educators to communicate effectively across different languages and cultures. This feature has opened up new possibilities for global collaboration and knowledge-sharing, and it has also made education more accessible to students from diverse backgrounds. Moreover, ChatGPT has become an essential tool for chatbots and virtual assistants used in education. Chatbots and virtual assistants have become increasingly popular in educational settings, as they can provide 24/7 support to students and answer their questions promptly. For example, new developments in AI can solve 81% of university-level mathematics questions and even come up with new questions.<ref name=":15">Drori, I., Zhang, S. X., Shuttleworth, R., Tang, L., Lu, A., Elizabeth, K. E., Liu, K. X., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N. K., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G. (2022). A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. ''Proceedings of the National Academy of Sciences of the United States of America'', ''119''(32). <nowiki>https://doi.org/10.1073/pnas.2123433119</nowiki></ref> With ChatGPT's ability to hold coherent conversations, these assistants can simulate human interactions and provide personalized support to students, thereby enhancing their learning experience.<ref name=":5" />


However, it is important to note that the implementation of AI in education is still in its early stages and there are many challenges that must be overcome, such as ensuring the privacy and security of student data and addressing ethical considerations and trust issues around the use of AI in the classroom.<ref name=":16">Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. ''Available at SSRN 4337484''.</ref><ref name=":17">Borenstein, J., & Howard, A. (2020). Emerging challenges in AI and the need for AI ethics education. ''AI and Ethics'', ''1''(1), 61–65. <nowiki>https://doi.org/10.1007/s43681-020-00002-7</nowiki></ref><ref name=":18">Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? ''International Journal of Educational Technology in Higher Education'', ''16''(1). <nowiki>https://doi.org/10.1186/s41239-019-0171-0</nowiki></ref> It also raises questions of transparency in regards to authorship credit and contributions of output by ChatGPT.<ref name=":19">Mohammad Hosseini, Lisa M. Rasmussen & David B. Resnik (2023) Using AI to write scholarly publications, Accountability in Research, DOI: [https://doi.org/10.1080/08989621.2023.2168535 10.1080/08989621.2023.2168535]</ref><ref name=":20">Huh, S. (2023). Recent Issues in Medical Journal Publishing and Editing Policies: Adoption of Artificial Intelligence, Preprints, Open Peer Review, Model Text Recycling Policies, Best Practice in Scholarly Publishing 4th Version, and Country Names in Titles. ''Neurointervention''. https://doi.org/10.5469/neuroint.2022.00493</ref> Furthermore, teachers fear that the AI would reduce their role to assistants and they also questioned the accuracy and reliability of the information generated by the system, as ChatGPT was criticized for its lack of understanding of meaning and content and its association with environmental racism and the interests of tech elites.<ref name=":10">Kim, N. J., & Kim, M. K. (2022). Teacher’s Perceptions of Using an Artificial Intelligence-Based Educational Tool for Scientific Writing. In ''Frontiers in Education'' (p. 142). Frontiers.</ref><ref name=":21">Williamson, B., Macgilchrist, F., & Potter, J. (2023). Re-examining AI, automation and datafication in education. ''Learning, Media and Technology'', ''48''(1), 1–5. https://doi.org/10.1080/17439884.2023.2167830</ref> The academic texts produced by ChatGPT are often irreversibly produced from ideas of researchers.<ref name=":22">Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. ''Minds and Machines'', ''30''(4), 681–694. <nowiki>https://doi.org/10.1007/s11023-020-09548-1</nowiki></ref> Even though many teachers are interested in using AI to improve their teaching practice, they also have concerns about the potential impact of AI on student learning, privacy, and job security.<ref name=":23">Lindner, A., Romeike, R., Jasute, E., & Pozdniakov, S. (2019). Teachers’ perspectives on artificial intelligence. In ''12th International conference on informatics in schools,“Situation, evaluation and perspectives”, ISSEP''.</ref><ref name=":24">Popenici, S.A.D., Kerr, S. Exploring the impact of artificial intelligence on teaching and learning in higher education. ''RPTEL'' 12, 22 (2017). <nowiki>https://doi.org/10.1186/s41039-017-0062-8</nowiki></ref>  
However, it is important to note that the implementation of AI in education is still in its early stages and there are many challenges that must be overcome, such as ensuring the privacy and security of student data and addressing ethical considerations and trust issues around the use of AI in the classroom.<ref name=":16">Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. ''Available at SSRN 4337484''.</ref><ref name=":17">Borenstein, J., & Howard, A. (2020). Emerging challenges in AI and the need for AI ethics education. ''AI and Ethics'', ''1''(1), 61–65. <nowiki>https://doi.org/10.1007/s43681-020-00002-7</nowiki></ref><ref name=":18">Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? ''International Journal of Educational Technology in Higher Education'', ''16''(1). <nowiki>https://doi.org/10.1186/s41239-019-0171-0</nowiki></ref>It also raises questions of transparency regarding authorship credit and contributions of output by ChatGPT.<ref name=":19">Mohammad Hosseini, Lisa M. Rasmussen & David B. Resnik (2023) Using AI to write scholarly publications, Accountability in Research, DOI: [https://doi.org/10.1080/08989621.2023.2168535 10.1080/08989621.2023.2168535]</ref><ref name=":20">Huh, S. (2023). Recent Issues in Medical Journal Publishing and Editing Policies: Adoption of Artificial Intelligence, Preprints, Open Peer Review, Model Text Recycling Policies, Best Practice in Scholarly Publishing 4th Version, and Country Names in Titles. ''Neurointervention''. https://doi.org/10.5469/neuroint.2022.00493</ref> Furthermore, teachers fear that the AI would reduce their role to assistants and they also questioned the accuracy and reliability of the information generated by the system, as ChatGPT was criticized for its lack of understanding of meaning and content and its association with environmental racism and the interests of tech elites.<ref name=":10">Kim, N. J., & Kim, M. K. (2022). Teacher’s Perceptions of Using an Artificial Intelligence-Based Educational Tool for Scientific Writing. In ''Frontiers in Education'' (p. 142). Frontiers.</ref><ref name=":21">Williamson, B., Macgilchrist, F., & Potter, J. (2023). Re-examining AI, automation and datafication in education. ''Learning, Media and Technology'', ''48''(1), 1–5. https://doi.org/10.1080/17439884.2023.2167830</ref> The academic texts produced by ChatGPT are often irreversibly produced from ideas of researchers.<ref name=":22">Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. ''Minds and Machines'', ''30''(4), 681–694. <nowiki>https://doi.org/10.1007/s11023-020-09548-1</nowiki></ref> Even though many teachers are interested in using AI to improve their teaching practice, they also have concerns about the potential impact of AI on student learning, privacy, and job security.<ref name=":23">Lindner, A., Romeike, R., Jasute, E., & Pozdniakov, S. (2019). Teachers’ perspectives on artificial intelligence. In ''12th International conference on informatics in schools,“Situation, evaluation and perspectives”, ISSEP''.</ref><ref name=":24">Popenici, S.A.D., Kerr, S. Exploring the impact of artificial intelligence on teaching and learning in higher education. ''RPTEL'' 12, 22 (2017). <nowiki>https://doi.org/10.1186/s41039-017-0062-8</nowiki></ref>[[File:Image chatGPT.png|thumb|Figure 1. ChatGPT]]


ChatGPT can be a game-changer in the field of education, providing personalized learning experiences, breaking down language barriers, and enhancing communication between educators and students. As technology continues to evolve, the potential for ChatGPT to transform education is significant, and it will be exciting to see how educators and students will leverage this model to create new opportunities for learning and knowledge-sharing. AI will help teachers in content delivery and other instructions, but in the future real life, human teachers might become obsolete.<ref name=":0">Alam, A. (2021). Possibilities and Apprehensions in the Landscape of Artificial Intelligence in Education. ''2021 International Conference on Computational Intelligence and Computing Applications (ICCICA)''. <nowiki>https://doi.org/10.1109/iccica52458.2021.9697272</nowiki></ref> But, while AI has the potential to greatly impact and improve education, it is important to approach its implementation with caution and careful consideration of its limitations and potential risks. This and the outlook and trust of teachers in academic education on AI in the classroom is what we will be researching. We will focus specifically on the course Engineering Design, a first year course mandatory to all TU/e students, where you go through a complete design process, from the first idea to the realization of the product. With the gathered information obtained from a survey and multiple interviews, we will create a deliverable useful for the course coordinators of engineering design, on how to handle ChatGPT in their course.  
ChatGPT can be a game-changer in the field of education, providing personalized learning experiences, breaking down language barriers, and enhancing communication between educators and students. As technology continues to evolve, the potential for ChatGPT to transform education is significant, and it will be exciting to see how educators and students will leverage this model to create new opportunities for learning and knowledge-sharing. AI will help teachers in content delivery and other instructions, but in the future real life, human teachers might become obsolete.<ref name=":0">Alam, A. (2021). Possibilities and Apprehensions in the Landscape of Artificial Intelligence in Education. ''2021 International Conference on Computational Intelligence and Computing Applications (ICCICA)''. <nowiki>https://doi.org/10.1109/iccica52458.2021.9697272</nowiki></ref> But, while AI has the potential to greatly impact and improve education, it is important to approach its implementation with caution and careful consideration of its limitations and potential risks. This and the outlook and trust of teachers and students in academic education on AI in the classroom is what will be researched. There will be focussed specifically on the course Engineering Design, a first year course mandatory to all TU/e students, where you go through a complete design process, from the first idea to the realization of the product. With the information obtained from a survey and multiple interviews, a deliverable useful for the course coordinators of engineering design will be created, on how to handle ChatGPT in their course.


==State of the Art==
==State of the Art==


===What is ChatGPT?===
===What is ChatGPT?===
ChatGPT is a new technology, but has been extensively researched, as it is such a noticeable new technology. ChatGPT is an acronym for Chat Generative Pre-trained Transformer and was created in November 2022. It is based on the OpenAI GPT-3 engine and has been fine-tuned by supervised and reinforcement learning technology; this means that the AI learns by having humans simulate artificial conversations with it and adapting its responses based on how accurately they reflect natural human dialogue. ChatGPT is also able to remember previously given prompts in the same conversation, making it a more personalized chatbot compared to its alternatives.<ref name=":2">Arimetrics (2022) ''What is ChatGPT.'' Retrieved from https://www.arimetrics.com/en/digital-glossary/chatgpt</ref> ChatGPT has a lot of functionalities, for example:
ChatGPT is a new technology, but has been extensively researched. ChatGPT is an acronym for Chat Generative Pre-trained Transformer and was created in November 2022. It is based on the OpenAI GPT-3 engine and has been fine-tuned by supervised and reinforcement learning technology; this means that the AI learns by having humans simulate artificial conversations with it and adapting its responses based on how accurately they reflect natural human dialogue. ChatGPT is also able to remember previously given prompts in the same conversation, making it a more personalized chatbot compared to its alternatives.<ref name=":2">Arimetrics (2022) ''What is ChatGPT.'' Retrieved from https://www.arimetrics.com/en/digital-glossary/chatgpt</ref> ChatGPT has a lot of functionalities, for example:


*Write and debug code, generate scripts and functions
*Write and debug code, generate scripts and functions


*Give detailed explanations on complex topics (answer test questions)
*Give detailed explanations on complex topics (answer test questions)
*Solve mathematical problems
*Write texts in different styles (write student essays)
*Write texts in different styles (write student essays)
*Compose music
*Compose music
Line 96: Line 97:
Another issue that comes up regarding academic integrity, is the unclarity on who the rightful author of the texts, ideas and inventions should be. Currently, there is no clear consensus on who should be considered the rightful author of AI-generated content. This is a complex issue that requires input from experts in moral philosophy, law, and computer science.<ref>Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). ''Ethics in Science and Environmental Politics'', ''21'', 17–23. <nowiki>https://doi.org/10.3354/esep00195</nowiki></ref>  
Another issue that comes up regarding academic integrity, is the unclarity on who the rightful author of the texts, ideas and inventions should be. Currently, there is no clear consensus on who should be considered the rightful author of AI-generated content. This is a complex issue that requires input from experts in moral philosophy, law, and computer science.<ref>Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). ''Ethics in Science and Environmental Politics'', ''21'', 17–23. <nowiki>https://doi.org/10.3354/esep00195</nowiki></ref>  


One more issue regards plagiarism when using ChatGPT. A study into this examined the accuracy and originality of scientific abstracts generated by ChatGPT compared to those produced by human experts. The study used a combination of artificial intelligence output detectors, plagiarism detectors, and blinded human reviewers to analyze the abstracts. The results showed that ChatGPT-generated abstracts had a higher similarity score with original abstracts than with other sources, but also revealed some cases of potential plagiarism. This suggests the need for caution when using AI-generated abstracts in scientific research. In the academic world, artificial intelligence models are gaining popularity due to their ability to enhance student engagement, collaboration, and accessibility. A paper authored by D. Cotton, explores the potential benefits and challenges of using AI in education. The models provide a platform for asynchronous communication, personalized and interactive assessments, and real-time grading and feedback. However, also according to D. Cotton there are concerns about academic integrity, particularly the possibility of plagiarism. With access to GPT-3, students could submit essays that are not their own work. Additionally, there are concerns about inequities in assessment, as students with access to GPT-3 have an advantage over those who do not. The article suggests solutions to combat these challenges, such as asking students to submit a draft before the final essay, set strict guidelines or monitor student work closely.<ref>Cotton, D., Cotton, P., & Shipway, J. R. (2023, January 10). Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT. https://doi.org/10.35542/osf.io/mrz8h</ref>
One more issue regards plagiarism when using ChatGPT. A study into this examined the accuracy and originality of scientific abstracts generated by ChatGPT compared to those produced by human experts. The study used a combination of artificial intelligence output detectors, plagiarism detectors, and blinded human reviewers to analyse the abstracts. The results showed that ChatGPT-generated abstracts had a higher similarity score with original abstracts than with other sources, but also revealed some cases of potential plagiarism. This suggests the need for caution when using AI-generated abstracts in scientific research. In the academic world, artificial intelligence models are gaining popularity due to their ability to enhance student engagement, collaboration, and accessibility. A paper, authored by D. Cotton, explores the potential benefits and challenges of using AI in education. The models provide a platform for asynchronous communication, personalized and interactive assessments, and real-time grading and feedback. However, also according to D. Cotton there are concerns about academic integrity, particularly the possibility of plagiarism. With access to GPT-3, students could submit essays that are not their own work. Additionally, there are concerns about inequities in assessment, as students with access to GPT-3 have an advantage over those who do not. The article suggests solutions to combat these challenges, such as asking students to submit a draft before the final essay, set strict guidelines or monitor student work closely.<ref>Cotton, D., Cotton, P., & Shipway, J. R. (2023, January 10). Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT. https://doi.org/10.35542/osf.io/mrz8h</ref>


Another paper explored the broader implications of ChatGPT for traditional models of education. The author argued that AI systems like ChatGPT have the potential to disrupt the traditional "banking model" of education, in which knowledge is transmitted from teacher to student. Instead, learners could access knowledge and information more directly through AI systems, which could democratize access to education and make it more inclusive. The author emphasizes the need for educators to adapt to this changing landscape and to shift their focus towards fostering critical thinking, creativity, and other skills that cannot be easily automated, and suggests that AI systems like ChatGPT have the potential to democratize access to education, making it more accessible and inclusive for learners around the world.<ref>Pimentel, C. Is ChatGPT a threat to education? For banking model of education, yes.</ref>
Another paper explored the broader implications of ChatGPT for traditional models of education. The author argued that AI systems like ChatGPT have the potential to disrupt the traditional "banking model" of education, in which knowledge is transmitted from teacher to student. Instead, learners could access knowledge and information more directly through AI systems, which could democratize access to education and make it more inclusive. The author emphasizes the need for educators to adapt to this changing landscape and to shift their focus towards fostering critical thinking, creativity, and other skills that cannot be easily automated, and suggests that AI systems like ChatGPT have the potential to democratize access to education, making it more accessible and inclusive for learners around the world.<ref>Pimentel, C. Is ChatGPT a threat to education? For banking model of education, yes.</ref>


Furthermore, a paper written by Rudolph also explores the implications of ChatGPT for both students and teachers. While ChatGPT offers personalized AI tutoring for students, it also has the potential to reduce the workload for teachers, especially with the use of automatic assessment tools. Furthermore, it can help teachers analyze their students' skills more easily. The paper gives a general overview of challenges and opportunities:<ref>ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? (2023). ''Journal of Applied Learning and Teaching'', ''6''(1). <nowiki>https://doi.org/10.37074/jalt.2023.6.1.9</nowiki></ref>
Furthermore, a paper written by Rudolph also explores the implications of ChatGPT for both students and teachers. While ChatGPT offers personalized AI tutoring for students, it also has the potential to reduce the workload for teachers, especially with the use of automatic assessment tools. Furthermore, it can help teachers analyse their students' skills more easily. The paper gives a general overview of challenges and opportunities:<ref>ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? (2023). ''Journal of Applied Learning and Teaching'', ''6''(1). <nowiki>https://doi.org/10.37074/jalt.2023.6.1.9</nowiki></ref>
{| class="wikitable"
{| class="wikitable"
|+Table 1. Challenges and opportunities of AI in education
|+Table 1. Challenges and opportunities of AI in education
Line 109: Line 110:
|Opportunity for teachers to improve/change their assessment and  teaching techniques.
|Opportunity for teachers to improve/change their assessment and  teaching techniques.
|-
|-
|ChatGPT doesn’t evaluate the relevance of the information, it just generates text that is an imitation  of what it has learned.
|ChatGPT doesn’t evaluate the relevance of the information, it just generates text that is an imitation  of what it has learned.
|''“ChatGPT allowed students to learn through experimentation and experience”''
|''“ChatGPT allowed students to learn through experimentation and experience”''
|}
|}
To look further into the fear of teachers that students will outsource all their work to ChatGPT, we will look another paper written by García-Peñalvo that reviews previous literature on ChatGPT. The most controversial issue with ChatGPT is the possibility of students using it as an easy solution to write essays without putting in the necessary effort. Because of this, they won’t acquire the needed knowledge for their course. However, the problem might not be the tool itself, but that the assessment techniques of educational institutions have become outdated However, the paper argues that prohibiting ChatGPT is not the way to go. Instead, teachers and students should learn how to use the tool to their advantage.<ref>García-Peñalvo, F. J. (2023). La percepción de la Inteligencia Artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. ''Education in the Knowledge Society'', ''24'', e31279. <nowiki>https://doi.org/10.14201/eks.31279</nowiki></ref>  
To look further into the fear of teachers that students will outsource all their work to ChatGPT, there will be looked to another paper written by García-Peñalvo that reviews previous literature on ChatGPT. The most controversial issue with ChatGPT is the possibility of students using it as an easy solution to write essays without putting in the necessary effort. Because of this, they won’t acquire the needed knowledge for their course. However, the problem might not be the tool itself, but that the assessment techniques of educational institutions have become outdated However, the paper argues that prohibiting ChatGPT is not the way to go. Instead, teachers and students should learn how to use the tool to their advantage.<ref>García-Peñalvo, F. J. (2023). La percepción de la Inteligencia Artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. ''Education in the Knowledge Society'', ''24'', e31279. <nowiki>https://doi.org/10.14201/eks.31279</nowiki></ref>  


As mentioned times before, ChatGPT has it obvious limitations. A controversial view in this paper written by Thorp, the author of the article is not necessarily worried about the use of ChatGPT in education as ‘it did well finding factual answers, but the scholarly writing still has a long way to go’. He thinks it pushes academics to design their courses in such a way that they are not easily solved by AI. The author is more worried for the influence of ChatGPT on the world of literature.<ref>Thorp, H. H. (2023). ChatGPT is fun, but not an author. ''Science'', ''379''(6630), 313. <nowiki>https://doi.org/10.1126/science.adg7879</nowiki></ref>  
As mentioned times before, ChatGPT has it obvious limitations. A controversial view in this paper written by Thorp, the author of the article is not necessarily worried about the use of ChatGPT in education as ‘it did well finding factual answers, but the scholarly writing still has a long way to go’. He thinks it pushes academics to design their courses in such a way that they are not easily solved by AI. The author is more worried about the influence of ChatGPT on the world of literature.<ref>Thorp, H. H. (2023). ChatGPT is fun, but not an author. ''Science'', ''379''(6630), 313. <nowiki>https://doi.org/10.1126/science.adg7879</nowiki></ref>  


While ChatGPT has its limitations, it has the potential to replace humans in routine tasks such as homework grading, potentially spelling the end of traditional essay writing assignments. "ChatGPT User Experience: Implications for Education" explores the user experience of ChatGPT and its potential impact on education. He also states that using AI tools to perform certain tasks ‘should be a part of the educational goals in the future’. The author concludes that teachers need to change their assignments to make it harder for students to use AI, but also notes that using AI in assignments to engage students in learning is a viable option. <ref>Zhai, X. (2022). ChatGPT User Experience: Implications for Education. ''Social Science Research Network''. <nowiki>https://doi.org/10.2139/ssrn.4312418</nowiki></ref>
While ChatGPT has its limitations, it has the potential to replace humans in routine tasks such as homework grading, potentially spelling the end of traditional essay writing assignments. "ChatGPT User Experience: Implications for Education" explores the user experience of ChatGPT and its potential impact on education. He also states that using AI tools to perform certain tasks ‘should be a part of the educational goals in the future’. The author concludes that teachers need to change their assignments to make it harder for students to use AI, but also notes that using AI in assignments to engage students in learning is a viable option. <ref>Zhai, X. (2022). ChatGPT User Experience: Implications for Education. ''Social Science Research Network''. <nowiki>https://doi.org/10.2139/ssrn.4312418</nowiki></ref>
Line 123: Line 124:


==Problem statement==
==Problem statement==
In the presented study, we investigate the influence of ChatGPT on students in reaching certain learning objectives. Therefore, our problem statement is as follows:
In the presented study, the influence of ChatGPT on students in reaching certain learning objectives is investigate. Therefore, the problem statement is as follows:


'''''How could the use of ChatGPT affect students in reaching the learning objectives of the course Engineering Design (4WBB0) at the TU/e?'''''
'''''How could the use of ChatGPT affect students in reaching the learning objectives of the course Engineering Design (4WBB0) at the TU/e?'''''
===Objectives:===
*Upcoming use of AI in academic settings,
*how to design this so that it will be accepted by teachers
*How AI can improve learning
*Acceptance towards AI in the classroom
*Looking at ChatGPT as main example
*Teachers at the TU/e, their thoughts about AI (in their courses)
===Users===
====Target group====
The target group of this project are the teachers of the TU/e who designed and teach the course engineering design (4WBB0) and students who followed this course. In this research the perspective of the teachers and students on the use of ChatGPT in this course is reviewed. The input of students will be used to get a better image of how students intend to use ChatGPT. The input of the teachers will be used to understand the learning objectives of the course engineering design and to determine the effect of ChatGPT on those learning objectives. The end product will be a deliverable useful for the course coordinators of engineering design.  
====Requirements====
Teachers want their students to deliver autonomous work, ChatGPT in academic settings can help with this. Teachers want to transfer their knowledge as effectively and efficiently as possible, using the current state of art technologies. AI technologies, like ChatGPT can help with this, but what teachers require is yet to be determined. The requirements will be determined by distributing surveys among students of the TU/e and conducting interviews with teachers, based on the survey results, in combination with literature studies. Those results will then be combined into a piece of well-considered advice for the teachers at the TU/e, consisting of guidelines for a course for the teachers of how to make exercises and tests for students that can benefit from using ChatGPT, but not be made solely with ChatGPT.  Which they will be able to use to guide them through the landscape of ChatGPT use in academic education and give them tools to keep transferring their knowledge to students.   


==Hypothesis==
==Hypothesis==
Line 247: Line 231:
The survey was sent to university students who had completed the Engineering Design course at TU/e, with the aim of gaining insights into their usage and perception of ChatGPT. The survey included questions related to the frequency of ChatGPT usage, perceived usefulness of the tool, and its impact on reaching the learning objectives of Engineering Design.
The survey was sent to university students who had completed the Engineering Design course at TU/e, with the aim of gaining insights into their usage and perception of ChatGPT. The survey included questions related to the frequency of ChatGPT usage, perceived usefulness of the tool, and its impact on reaching the learning objectives of Engineering Design.


We attended a panel discussion hosted by the TU/e about ChatGPT in Engineering Education. From this and background literature a series of interviews were conducted with both a professor from the Engineering Design department and frequent users of ChatGPT: which will be called experts. The interviews aimed to provide a deeper understanding of the participants' opinions and experiences with ChatGPT, specifically in the context of Engineering Design. The interviews with the student experts aimed to capture their motivation for using ChatGPT, how they use the tool, and their perception of its impact on the Engineering Design learning goals. The interview with the professor aimed to capture the impact they believe ChatGPT has on the learning objectives, and their overall opinion on the use of ChatGPT.
A panel discussion hosted by the TU/e about ChatGPT in Engineering Education was attended. From this and background literature, a series of interviews were conducted with both a professor from the Engineering Design department and frequent users of ChatGPT: which will be called experts. The interviews aimed to provide a deeper understanding of the participants' opinions and experiences with ChatGPT, specifically in the context of Engineering Design. The interviews with the student experts aimed to capture their motivation for using ChatGPT, how they use the tool, and their perception of its impact on the Engineering Design learning goals. The interview with the professor aimed to capture the impact they believe ChatGPT has on the learning objectives, and their overall opinion on the use of ChatGPT.


Together, the survey and interviews aimed to obtain a comprehensive understanding of the participants' opinions regarding the use of ChatGPT in the Engineering Design course and its impact on achieving the learning objectives within this course.
Together, the survey and interviews aimed to obtain a comprehensive understanding of the participants' opinions regarding the use of ChatGPT in the Engineering Design course and its impact on achieving the learning objectives within this course.


===Survey===
===Survey===
The survey consists of questions about the participants' general knowledge and experience about ChatGPT, and how this tool could affect them reaching the learning goals of Engineering Design. The questions of the survey can be found in appendix 2.  
The survey consists of questions about the participants' general knowledge and experience about ChatGPT, and how this tool could affect them reaching the learning goals of Engineering Design. The questions from the survey can be found in appendix 2.  


=====Participants=====
=====Participants=====
The survey was distributed to students who are enrolled at the TU/e and who have finished the course Engineering Design, which are 2nd year and higher students. The participants were obtained by asking students in the social circles of the participants, and spreading a link in text group chats. Furthermore we also aimed for snowball sampling. There were a total of 39 responses.
The survey was distributed to students who are enrolled at the TU/e and who have finished the course Engineering Design, which are 2nd and higher year students. The participants were obtained by asking students in the social circles of the researchers, and spreading a link in text group chats. There were a total of 39 responses.


=====Procedure=====
=====Procedure=====
The outline of the survey was as follows (see appendix 2): a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the affects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals.
The outline of the survey was as follows (see appendix 2): a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals.
===Panel discussion===
===Panel discussion===
On 23rd of march the TU/e hosted a panel discussion about ChatGPT in Engineering Education. This panel included an associate professor in the department of philosophy and ethics who is doing a study on disruptive technologies, an assisting professor in the department of philosophy and ethics who is doing a study on the philosophy of AI, a Data Science master student, a professor in Computer Science,  a Complied Physics master student, a professor in the department of Mechanical Engineering, the program manager of Education Innovation and Technology who is also a advisor for the examination committee here on the TU/e and the host CTO of FruitPunch AI and former Computer Science student here at the TU/e. We observed this discussion and took notes of the things that were interesting for us to use when conducting the interviews later on.
On 23rd of march, the TU/e hosted a panel discussion about ChatGPT in Engineering Education. This panel included an associate professor in the department of philosophy and ethics who is doing a study on disruptive technologies, an assisting professor in the department of philosophy and ethics who is doing a study on the philosophy of AI, a Data Science master student, a professor in Computer Science, a Complied Physics master student, a professor in the department of Mechanical Engineering, the program manager of Education Innovation and Technology who is also an advisor for the examination committee here on the TU/e and the host CTO of FruitPunch AI and former Computer Science student here at the TU/e. This discussion was observed and notes were taken of the things that were interesting to use when conducting the interviews later on.
===Interview with ChatGPT experts===
===Interview with ChatGPT experts===
Student experts who frequently use ChatGPT were interviewed to gain a better understanding of how and why they use ChatGPT, and how this usage could impact achieving the learning goals of Engineering Design.
Student experts who frequently use ChatGPT were interviewed to gain a better understanding of how and why they use ChatGPT, and how this usage could impact achieving the learning goals of Engineering Design.


=====Participants=====
=====Participants=====
The focus was on university students who were all enrolled at the TU/e and had successfully completed the Engineering Design course. A total of five participants were recruited, one female and four males, ranging from the ages 20 to 23 (M<sub>age</sub>= 21, SD = 1.79, 20% female). The participants were sampled by the convenience sampling method, all the participants had to be students at the University of Eindhoven, and they should have followed the course Engineering Design, since this was a crucial part for our research.  
The focus was on university students who were all enrolled at the TU/e and had successfully completed the Engineering Design course. A total of five participants were recruited, one female and four males, ranging from the ages 20 to 23 (M<sub>age</sub>= 21, SD = 1.79, 20% female). The participants were sampled by the convenience sampling method, all the participants had to be students at the University of Eindhoven, and they should have taken the course Engineering Design, since this was a crucial part for our research.  


=====Procedure=====
=====Procedure=====
The outline of the interview was as follows: a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the affects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals. We asked open questions that were mentioned in the interview script, see appendix 3. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analyzed carefully, which gave a certain conclusion of the interview.
The outline of the interview was as follows: a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals. Open questions were asked that were mentioned in the interview script, see appendix 3. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analysed carefully, which gave a certain conclusion of the interview.


===Interview with university professor===
===Interview with university professor===
Line 274: Line 258:


=====Participants=====
=====Participants=====
The participant for this interview was Joris Remmers, an associate professor of Mechanical Engineering, whose role within Engineering Design is to organize assessments and set up the organizational structure of it. He is responsible for the assessment setting and rubrics.
The participant for this interview was Joris Remmers, an associate professor of Mechanical Engineering, whose role within Engineering Design is to organize assessments and set up the organizational structure of it. He is responsible for the assessment setting and rubrics.[[File:Joris Remmers.png|thumb|Figure 2. Joris Remmers]]<br />


=====Procedure=====
=====Procedure=====
The outline of the interview was as follows: a brief section about his general knowledge of ChatGPT, his opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the affects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participant, and the latter to measure the effects of ChatGPT on achieving certain learning goals. We asked open questions that were mentioned in the interview script, see appendix <s>4</s>. If the participant did not give a clear answer, we asked follow-up questions to get more in-depth answers. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analyzed carefully, which gave a certain conclusion of the interview.
The outline of the interview was as follows: a brief section about his general knowledge of ChatGPT, his opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participant, and the latter to measure the effects of ChatGPT on achieving certain learning goals. Open questionswere asked that were mentioned in the interview script, see appendix <s>4</s>. If the participant did not give a clear answer, follow-up questions were asked to get more in-depth answers. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analysed carefully, which gave a certain conclusion of the interview.


===Thematic analysis===
===Thematic analysis===
All the interviews have been analyzed with the use of thematic analysis. Thematic analysis makes use of finding patterns in the answers to the interview questions. The thematic analysis was conducted via Dedoose<ref>“Home | Dedoose”. <nowiki>https://www.dedoose.com/</nowiki></ref>. Codes for themes in the interview were made based on the apparent feelings the interviewees have about ChatGPT and its use. Three main groups were established: Features, what interviewees use it for and why they use it; feelings, what the general feeling of interviewees is about ChatGPT; and warnings, what interviewees think is a drawback of ChatGPT.  
All the interviews have been analysed with the use of thematic analysis. Thematic analysis makes use of finding patterns in the answers to the interview questions. The thematic analysis was conducted via Dedoose<ref>“Home | Dedoose”. <nowiki>https://www.dedoose.com/</nowiki></ref>. Codes for themes in the interview were made based on the apparent feelings the interviewees have about ChatGPT and its use. Three main groups were established: Features, what interviewees use it for and why they use it; feelings, what the general feeling of interviewees is about ChatGPT; and warnings, what interviewees think are drawbacks of ChatGPT. These results were used to more easily interpret and compare the answers given in the interviews.  
==Results==
==Results==


===User procifiency===
===User proficiency<!-- add some more text probs -->===
[insert table/visualisation on the relation between use and proficiency (learning goal attainment?)
[[File:Jitterplot1.png|none|thumb|451x451px|Figure 3. Learning goal obtainability by user proficiency]]
 
To get a clearer image of the results of the interviews, it is wise to compare the difference in answers between students who were very proficient in using ChatGPT an students who used it now and then for more superficial use. As can be seen in the jitter plot above (figure 3) there is no clear correlation to be seen in the data.  
To get a clear image of the results of the interviews it is wise to compare the difference in answers between students who were very proficient in using ChatGPT an students who used it now and then for more superficial use.
===Survey analysis===
===Survey analysis===
In the survey we started with some general questions. All participants have heard of ChatGPT before, however 13% do not know its capabilities. When asked to name some capabilities of ChatGPT, students mention the generation of text in response to prompts given, they mention that it gives answers to questions, it helps with writing essays and writing code. The emphasis is mostly on the fact that you can do a lot with ChatGPT, you can ask it basically anything. Quite some students in the survey have not used ChatGPT before, specifically 33% of the participants. Most students have used ChatGPT either just once or twice, or weekly. Two participants use it daily, and four participants use it monthly.  
In the survey we started with some general questions. All participants have heard of ChatGPT before, however 13% do not know its capabilities. When asked to name some capabilities of ChatGPT, students mention the generation of text in response to prompts given, they mention that it gives answers to questions, it helps with writing essays, and writing code. The emphasis is mostly on the fact that you can do a lot with ChatGPT, you can ask it basically anything. Quite a few students in the survey have not used ChatGPT before, specifically 33% of the participants. Most students have used ChatGPT either just once or twice, or weekly. Two participants use it daily, and four participants use it monthly.  


In the survey we have asked students what they have used ChatGPT for. The results are very varied: students use it for a lot of different things. Things that students use it for most frequently are just fooling around with it, writing parts of papers, or general academic work. Students also use it for writing emails and summaries, and for writing code or finding errors and solutions in already written code. Several students also mention that they use ChatGPT for answering questions and queries in general, using it more as a search engine. Finally students also mention using it for getting design ideas, and other creative ideas. For an overview look at table 3.
In the survey we have asked students what they have used ChatGPT for. The results are very varied: students use it for a lot of different things. Things that students use it for most frequently are just fooling around with it, writing parts of papers, or general academic work. Students also use it for writing emails and summaries, and for writing code or finding errors and solutions in already written code. Several students also mention that they use ChatGPT for answering questions and queries in general, using it more as a search engine. Finally students also mention using it for getting design ideas, and other creative ideas. For an overview look at table 3.
Line 338: Line 321:
|9x
|9x
|}
|}
We asked the students to rate the experience with ChatGPT, and the students rated ChatGPT on average 4 out of 5 stars. We also asked them to rate it as an academic tool, and they rated it 3.5 stars out of 5. These are both pretty high ratings, and the students seem to be happy with ChatGPT and how it performs. 65% of students mention that they have not used ChatGPT for academic purposes, however 35% have used it, and for various different courses.  
Students were asked to rate the experience with ChatGPT, and on average, the students rated ChatGPT 4 out of 5 stars. We also asked them to rate it as an academic tool, and they rated it 3.5 stars out of 5. These are both pretty high ratings, and the students seem to be happy with ChatGPT and its performance. 65% of students mention that they have not used ChatGPT for academic purposes, however 35% have used it, and for various different courses.  


We also asked participants how they felt about the ownership of output created by ChatGPT. Only 31% felt that usage of ChatGPT should be considered their own work. When asked when the use of ChatGPT should be considered plagiarism, students mention that when it is copied directly from ChatGPT, when it is presented as their own work, and when it does entire exercises for you, they think it should be considered as plagiarism.
Participants were also asked how they felt about the ownership of output created by ChatGPT. Only 31% felt that usage of ChatGPT should be considered their own work. When asked when the use of ChatGPT should be considered plagiarism, students mention that when it is copied directly from ChatGPT, when it is presented as their own work, and when it does entire exercises for you, they think it should be considered as plagiarism.


In the survey we asked questions specifically relating to the course Engineering Design. 64% of the participants feel that ChatGPT could be a good addition to the course. When asked with what they think it can help, students mention getting design ideas quite frequently. Furthermore they also mention getting help with electronics, ChatGPT being a ‘better google’, help with writing the report and help with coding. Students do mention that they believe for the course creative thinking is required, which they feel ChatGPT is not capable of. Students are divided on whether they think ChatGPT should be used in the course, some say it would be a great addition, others fear it will take away from the learning process or they do not think ChatGPT will add anything useful.  
In the survey questions specifically relating to the course Engineering Design were asked. 64% of the participants feel that ChatGPT could be a good addition to the course. When asked how they think it can help, students mention getting design ideas quite frequently. Furthermore they also mention getting help with electronics, ChatGPT being a ‘better Google’, help with writing the report and help with coding. Students do mention that they believe for the course creative thinking is required, which they feel ChatGPT is not capable of. Students are divided on whether they think ChatGPT should be used in the course, some say it would be a great addition, others fear it will take away from the learning process or they do not think ChatGPT will add anything useful.  


In the survey we provided several statements, and asked the participants on their feelings about this.  
In the survey we provided several statements, and asked the participants their feelings about these.  


The first statement: ‘I see ChatGPT as having/could have had a positive influence on my engineering design project.’, most students tend to agree (33%). However students are divided on this topic, as 20,5% disagrees. This divide was present in the previous questions as well, where it could be seen students have differing opinions regarding ChatGPT.
The first statement: ‘I see ChatGPT as having/could have had a positive influence on my engineering design project.’, most students tend to agree with (33%). However, students are divided on this topic, as 20.5% disagrees. This divide was present in the previous questions as well, where it could be seen students have differing opinions regarding ChatGPT.


The second statement: ‘I would have seen ChatGPT as a useful tool for my engineering design project.’, even more students agree with, namely 41%. There is also a large percentage of students who are undecided on this matter, namely 25%.  
The second statement: ‘I would have seen ChatGPT as a useful tool for my engineering design project.’, even more students agree with, namely 41%. There is also a significant percentage of students who are undecided on this matter, namely 25%.  


The third statement: ‘I see ChatGPT as a shortcut for tedious tasks I would have/had to perform in the course engineering design.’, most students tend to agree with, 33% of them, and 20% even say they strongly agree. However, on the opposite side, 18% also say they disagree. Most students do see ChatGPT as a shortcut, for helping them with tasks that would have otherwise cost them much more time.
The third statement: ‘I see ChatGPT as a shortcut for tedious tasks I would have/had to perform in the course engineering design.’, the majority of students tend to agree with, 33% of them, and 20% even say they strongly agree. However, on the opposite side, 18% also say they disagree. Most students do see ChatGPT as a shortcut, for helping them with tasks that would have otherwise cost them much more time.


For the fourth statement: ‘I will use/would have used ChatGPT in the course engineering design regularly.’, students very often disagree with. 28% disagrees with this statement, and 28% even strongly disagrees. Only 23% of students agree with this statement. Here it shows as although most students do seem to see some usefulness in ChatGPT, they do not think they will use it that often. It is most likely not something they will integrate into their toolbox and use as a first inclination.
For the fourth statement: ‘I will use/would have used ChatGPT in the course engineering design regularly.’, students very often disagree with. 28% disagrees with this statement, and 28% even strongly disagrees. Only 23% of students agree with this statement. Here it shows as although most students do seem to see some usefulness in ChatGPT, they do not think they will use it that often. It is most likely not something they will integrate into their toolbox and use as a first inclination.
Line 358: Line 341:
Overall students seem to have mixed feelings about ChatGPT and the use of ChatGPT for the course engineering design. Students seem to either see the use in it, and think it is a good addition, or they do not see the use in it, and do not see themselves using it. Students also do not have that much experience with ChatGPT yet, and have not gotten the full grasp of its winnings but also its limitations.
Overall students seem to have mixed feelings about ChatGPT and the use of ChatGPT for the course engineering design. Students seem to either see the use in it, and think it is a good addition, or they do not see the use in it, and do not see themselves using it. Students also do not have that much experience with ChatGPT yet, and have not gotten the full grasp of its winnings but also its limitations.


In the survey we also asked the participants to judge the learning goals of the course Engineering Design on how likely they think these goals can be reached with ChatGPT. In the graph beneath the answers can be seen.[[File:Survey learning goals.png|none|thumb|Figure 1. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"]]From this it can be seen that students see ChatGPT being the most useful for formulating a design goal and writing the report. These are both text entries, which could be done with the use of ChatGPT, which students seems to recognize. Furthermore defining the functional and technical specifications and executing a generic design process students also seem to think can be done . The first can be directly asked in ChatGPT. The latter is not something you can ask ChatGPT, however students do seem to think they can reach this learning goal with the help of ChatGPT. Students seem to think that the learning goal make a final design choice between a number of concepts has the least potential to be done with ChatGPT. They also do not think selecting a number of design concepts from an extensive list of realization possibilities and developing a detailed design that meets the specifications are goals that can be reached with the use of ChatGPT.  
In the survey we also asked the participants to judge the learning goals of the course Engineering Design on how likely they think these goals can be reached with ChatGPT. In figure 4 the answer distribution can be seen.[[File:Survey learning goals.png|none|thumb|Figure 4. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"]]From this it can be seen that students see ChatGPT being the most useful for formulating a design goal and writing the report. These are both text entries, which could be done with the use of ChatGPT, which students seem to recognize. Furthermore, students also seem to think that defining the functional and technical specifications and executing a generic design process can be done. The former can be directly provided by ChatGPT, but the latter is not something you can ask ChatGPT. However, students do seem to think they can reach this learning goal with the help of ChatGPT. Students seem to think that the learning goal make a final design choice between a number of concepts has the least potential to be done with ChatGPT. They also do not think selecting a number of design concepts from an extensive list of realization possibilities and developing a detailed design that meets the specifications are goals that can be reached with the use of ChatGPT.  


Still, students think that quite some learning goals could benefit from or be done with ChatGPT. Here it seems students would think that ChatGPT could be useful for the course.
Still, students think that quite some learning goals could benefit from or be done with ChatGPT. Here it seems students would think that ChatGPT could be useful for the course.
Line 365: Line 348:
On 23rd of march the TU/e hosted a panel discussion about ChatGPT in Engineering Education. A lot of important questions came up; How might it impact workload of teachers? Is the implementation of ChatGPT going to deskill students? How is the TU/e going to check for plagiarism? And is this going to cause data/privacy issues?
On 23rd of march the TU/e hosted a panel discussion about ChatGPT in Engineering Education. A lot of important questions came up; How might it impact workload of teachers? Is the implementation of ChatGPT going to deskill students? How is the TU/e going to check for plagiarism? And is this going to cause data/privacy issues?


During the discussion, the participants shared their thoughts on ChatGPT and its impact on education. They agreed that while it is exciting technology, it can also be disruptive, and its potential for innovation is a cause for concern for education. However, they also recognized that human adaptability is strong, and technology like ChatGPT should be given a place.
During the discussion, the participants shared their thoughts on ChatGPT and its impact on education. They agreed that while it is exciting technology, it can also be disruptive and cause a concern for education. However, they also recognized that human adaptability is strong, and technology like ChatGPT should be given a place. None of the panel members had a clear answer to what this place should be, since it is so hard to check how much a student actually learns when they use ChatGPT.


The participants emphasized the importance of clear communication from TU/e and its teachers about what they teach and their purpose. They also discussed the need to figure out how to view technology like ChatGPT and involve students in the discussion. This challenge presents an opportunity to consider what students really need to learn for their future.
The participants emphasized the importance of clear communication from TU/e and its teachers about what they teach and their purpose. They also discussed the need to figure out how to view technology like ChatGPT and involve students in the discussion. This challenge presents an opportunity to consider what students really need to learn for their future.
Line 374: Line 357:
A very concise summary of the interviews with the five ChatGPT experts and professor can be seen in tables 4 and 5 below: these tables depict whether these participants think it would be likely to reach the mentioned learning goal with the use of ChatGPT.
A very concise summary of the interviews with the five ChatGPT experts and professor can be seen in tables 4 and 5 below: these tables depict whether these participants think it would be likely to reach the mentioned learning goal with the use of ChatGPT.


We consulted five student experts and gathered their answers in the appendix table XX, which we used to create table 4. This table shows the answers that most experts agreed on. We considered an answer to be common if at least four out of five experts agreed on it, which corresponds to 80% agreement. For example, if four or more experts answered 'yes' to a question, we assigned 'yes' as the common answer for that learning objective. However, if the experts' answers were too different, meaning that less than four experts agreed, we marked the result as 'inconclusive'.
We consulted five student experts and gathered their answers in the appendix table 8, which we used to create table 4. This table shows the answers that most experts agreed on. We considered an answer to be common if at least four out of five experts agreed on it, which corresponds to 80% agreement. For example, if four or more experts answered 'yes' to a question, we assigned 'yes' as the common answer for that learning objective. However, if the experts' answers were too different, meaning that less than four experts agreed, we marked the result as 'inconclusive'.


The results of the expert interviews and professor interview are compared in table 6.
The results of the expert interviews and professor interview are compared in table 6.
Line 519: Line 502:
!Professor
!Professor
!Agreement
!Agreement
!Hypothesis
|-
|-
|Execute a generic design process
|Execute a generic design process
Line 524: Line 508:
|Yes
|Yes
|
|
|No
|-
|-
|Formulate a design goal
|Formulate a design goal
Line 529: Line 514:
|No
|No
|
|
|Yes
|-
|-
|Define the functional and technical specifications
|Define the functional and technical specifications
Line 534: Line 520:
|Yes
|Yes
|✓
|✓
|Yes
|-
|-
|Generate an elaborate list of realization possibilities for the different functions of the design
|Generate an elaborate list of realization possibilities for the different functions of the design
Line 539: Line 526:
|No
|No
|
|
|Yes
|-
|-
|Select a number of design concepts from an extensive list of realization possibilities
|Select a number of design concepts from an extensive list of realization possibilities
Line 544: Line 532:
|Yes
|Yes
|
|
|Yes
|-
|-
|Make a final design choice between a number of concepts
|Make a final design choice between a number of concepts
Line 549: Line 538:
|No
|No
|✓
|✓
|No
|-
|-
|Develop a detailed design that meets the specifications
|Develop a detailed design that meets the specifications
Line 554: Line 544:
|Yes
|Yes
|✓
|✓
|No
|-
|-
|Develop and execute a test plan for the prototype
|Develop and execute a test plan for the prototype
Line 559: Line 550:
|Yes
|Yes
|
|
|No
|-
|-
|Evaluate a prototype based on test results and give an advice for redesign
|Evaluate a prototype based on test results and give an advice for redesign
Line 564: Line 556:
|Yes
|Yes
|
|
|Yes
|-
|-
|Reflect on the design and on the design process
|Reflect on the design and on the design process
Line 569: Line 562:
|No
|No
|
|
|No
|-
|-
|Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design
|Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design
Line 574: Line 568:
|Yes
|Yes
|
|
|Yes
|}
|}


====Thematic analysis====
====Thematic analysis====
In table 'Codes for the themes' all the found codes and there descriptions are given.
In table 7 'Codes for the themes' all the found codes and there descriptions are given.
{| class="wikitable"
{| class="wikitable"
|+Table 7. Codes for the themes
|+Table 7. Codes for the themes
!Overarching theme
!Overarching theme
!Code
!Code
!Discription
!Description
|-
|-
| -
| -
Line 672: Line 667:
|When it was mentioned that ChatGPT created a gap between the user and its goal.
|When it was mentioned that ChatGPT created a gap between the user and its goal.
|}
|}
These codes are applied to the interviews when their description matches what is mentioned. What follows from it is thematic analysis code occurrence table. In the table below you can find an overview of the thematic analyses of the interviews.<br />[[File:Code application in Interviews.png|center|thumb|800x800px|Figure 2. Thematic analysis code occurrence from the interviews conducted with students who are proficient users of ChatGPT.]]
These codes are applied to the interviews when their description matches what is mentioned. What follows from it is thematic analysis code occurrence table. Below (figure 5) you can find an overview of the thematic analyses of the interviews.<br />[[File:Code application in Interviews.png|center|thumb|800x800px|Figure 5. Thematic analysis code occurrence from the interviews conducted with students who are proficient users of ChatGPT.]]
 


The thematically analysed interviews can be found on the following page: https://surfdrive.surf.nl/files/index.php/s/drOP7BwOUh8bzKs
With the password being: 0LAUK0


==Discussion==
==Discussion==
Line 680: Line 679:


====Surveys====
====Surveys====
Based on the survey results, it can be interpreted that university students think that ChatGPT is a well-known and versatile tool that is primarily used for generating text in response to prompts, answering questions, and assisting with writing tasks such as essays and code. The survey indicates that students value the fact that ChatGPT can be used for a wide range of tasks and that it is highly flexible in its capabilities. It also suggests that students are generally familiar with ChatGPT and its capabilities, although some may not know its full range of abilities.
Based on the survey results, it becomes clear that university students think that ChatGPT is a very helpful tool that is mainly being used for answering questions, explaining things, and helping with writing essays or code. This shows that students highly value the fact that ChatGPT can be used for a wide range of different tasks. Although many students are familiar with ChatGPT's capabilities, some may not know its full potential.


The overall experience with ChatGPT is rated positively, with high ratings for general use and academic use. However, students are divided on whether ChatGPT should be used in the course of Engineering Design, with some students seeing the potential benefits, while others are more hesitant. Students believe ChatGPT can be a useful tool for certain aspects of the Engineering Design course. Specifically, they see the most potential for ChatGPT in tasks that involve text, such as formulating design goals and writing reports. Additionally, they believe that ChatGPT can assist in defining functional and technical specifications, as well as executing a generic design process.
The overall experience with ChatGPT is considered positive. However, students are not in agreement on whether ChatGPT should be used in the course Engineering Design, with some students seeing the useful possibilities, while others are less enthusiastic. The advocates for the chatbot believe that ChatGPT can be used for certain aspects; specifically, tasks that involve text and writing, such as formulating design goals and writing reports. Additionally, they believe that ChatGPT can help with defining functional and technical specifications, as well as executing a generic design process.


However, students are less convinced that ChatGPT can help with more complex tasks, such as making a final design choice between a number of concepts or selecting a number of design concepts from an extensive list of realization possibilities. They also do not believe that ChatGPT can help with developing a detailed design that meets specifications. Another concern expressed by students is the issue of ownership of the output created by ChatGPT. A relatively small proportion of students believe that usage of ChatGPT should be considered their own work, and students tend to view direct copying from ChatGPT and presenting it as their own as plagiarism.
On the other hand, students are less inclined to use ChatGPT for more complex tasks, such as making a final design choice or selecting a number of design concepts from an extensive list of realization possibilities. They also do not believe that ChatGPT can help with developing a detailed design that meets specifications. Another point the students make is about whether using ChatGPT could be considered as plagiarism. Merely a small portion believe that ChatGPT's output should be considered their own original work, while the majority would classify directly copying ChatGPT as plagiarism.  


Overall, it appears that students have a realistic understanding of what ChatGPT can and cannot do, and are aware of its limitations. Despite this, they still see potential for its use in certain aspects of the Engineering Design course, suggesting that it could be a valuable tool to complement their learning experience.
Overall, it seems that students have a realistic understanding of what ChatGPT can and cannot do, and are aware of its limitations. Despite this, they still see potential for its use in certain aspects of Engineering Design, suggesting that it could be a valuable tool to help develop their skills.


====Interviews ChatGPT expert====
====Interviews ChatGPT expert====
Through thematic analysis, several general notions emerged regarding the usage of ChatGPT among the interviewed students. The primary reasons for using ChatGPT were identified as gaining information, receiving writing assistance, and saving time. Other notable purposes included using it for creativity, academic pursuits, code verification, and guidance. One interviewee even remarked that they used ChatGPT as a substitute for Google search.  
Through thematic analysis, several things can be said regarding the usage of ChatGPT among the interviewed students who are ChatGPT experts. They mentioned that their main reasons for using ChatGPT were gaining information, receiving writing feedback, and to save time, since ChatGPT is a very efficient tool. Other reasons are using it for creativity, academic purposes, and programming help. One interviewee even mentioned that they used ChatGPT as a substitute for Google search.  


Interestingly, all interviewees acknowledged certain drawbacks of using ChatGPT. The most frequently mentioned disadvantage was its limited functionality, which emerged as the predominant theme. This concern was particularly relevant in academic settings, as interviewees believed that ChatGPT could only handle "straightforward stuff" and that it should be used cautiously. One interviewee even mentioned that ChatGPT "hallucinates" and is not up to date, referring to the fact that ChatGPT's database only reaches up until 2021.
Interestingly, all interviewees acknowledged the pitfalls of using ChatGPT. The most frequently mentioned disadvantage was its limited functionality, which was a predominant theme. This drawback was especially apparent in academic settings, as interviewees believed that ChatGPT could only handle "straightforward stuff" and that it should be used cautiously. One interviewee even mentioned that ChatGPT "hallucinates" and is not up to date, referring to the fact that ChatGPT's database only reaches up until 2021.


Based on the responses from the student experts who were asked whether they think the learning objectives could be achieved with the help of ChatGPT, it appears that the students believe that ChatGPT has the most potential to assist with the learning objectives of formulating a design goal and writing a design report. They also believe that ChatGPT could be helpful in defining functional and technical specifications, and in executing a generic design process. However, the students do not believe that ChatGPT would be particularly useful for selecting a number of design concepts from a list of realization possibilities, developing a detailed design that meets specifications, or evaluating a prototype based on test results and giving advice for redesign.
Based on the responses from the student experts who were asked whether they think the learning objectives could be achieved with the help of ChatGPT, it turns out that the experts have differing opinions. Collectively, they believe that ChatGPT has the most potential to help with the learning objectives of formulating a design goal and writing a design report, as these goals focus more on the writing aspect. They also believe that ChatGPT could be helpful in defining functional and technical specifications, and in executing a generic design process. However, the students do not believe that ChatGPT would be particularly useful for selecting a number of design concepts from a list of realization possibilities, developing a detailed design that meets specifications, or evaluating a prototype based on test results and giving advice for redesign.
 
Interestingly, the students seem to have differing opinions on the feasibility of certain learning objectives with ChatGPT. For example, while some students think that defining functional and technical specifications would be "hard but possible" with ChatGPT, others believe it would be "good for this." Similarly, while some students believe that executing a generic design process is achievable with ChatGPT, others think it is "not doable."


Overall, the students seem to think that ChatGPT could be a helpful tool for certain aspects of the design process, but it may not be suitable for all tasks. It is important to note that these opinions are based solely on the perspectives of the five interviews students, and may not be representative of all potential users of ChatGPT or all design students.
Overall, the students seem to think that ChatGPT could be a helpful tool for certain aspects of the design process, but it may not be suitable for all tasks. It is important to note that these opinions are based solely on the perspectives of the five interviews students, and may not be representative of all potential users of ChatGPT or all design students.


====Interview university professor====
====Interview university professor====
The interviewee was overall positive about his experience with ChatGPT and even encourages his students to use it. The main benefits were: improving students' report writing, using it as a more efficient search engine, and helping with speeding up doing ordinary tasks, such as googling for literature. However, according to him, the main drawback of the chatbot is that it is a very linguistic tool. He emphasized the fact that ChatGPT was not creative enough and could not generate innovative ideas due to its limited database.  
The professor, Joris Remmers, was overall positive about his experience with ChatGPT and even encourages his students to use it. The main benefits he mentioned were: improving students' report writing, using it as a more efficient search engine, and helping with speeding up doing ordinary tasks, such as googling for literature. However, according to him, the main drawback of the chatbot is that it is a very linguistic tool. He emphasized the fact that ChatGPT was not creative enough and could not generate innovative ideas due to its limited database.  


These statements are backed up by his thoughts on whether it would be likely to reach the learning objectives of Engineering Design with the use of ChatGPT. The professor believes that ChatGPT can be useful in certain aspects of the engineering design process, but not all. ChatGPT can assist students in executing a generic design process and defining the functional and technical specifications, as these parts require objective knowledge and can be looked up. ChatGPT can also be used to develop and execute a test plan for the prototype and to evaluate a prototype based on test results, particularly if the prototype is software design.  
These statements are backed up by his thoughts on whether it would be likely to reach the learning objectives of Engineering Design with the use of ChatGPT. He believes that ChatGPT can be helpful in some, but definitely not all objectives. ChatGPT can help students in executing a generic design process and defining the functional and technical specifications, as these parts require objective knowledge and can be looked up. ChatGPT can also be used to develop and execute a test plan for the prototype and to evaluate a prototype based on test results, particularly if the prototype is software design.  


However, ChatGPT is not suitable for some of the most creative parts of the process, such as formulating a design goal and generating an elaborate list of realization possibilities for the different functions of the design. ChatGPT can help students to select a number of design concepts from an extensive list of realization possibilities by following strict procedures, but students ultimately need to reflect on their own preferences to make a final design choice between a number of concepts. ChatGPT can also be used to structure the concepts for the detailed design, but students need to do the thinking, combining results, solving equations, and doing the actual work.
However, ChatGPT is not suitable for some of the most creative parts of the process, such as formulating a design goal and generating an elaborate list of realization possibilities for the different functions of the design. ChatGPT can help students to select a number of design concepts from an extensive list of realization possibilities by following strict procedures, but students ultimately need to reflect on their own preferences to make a final design choice between a number of concepts. ChatGPT can also be used to structure the concepts for the detailed design, but students need to do the thinking, combining results, solving equations, and doing the actual work.


Finally, ChatGPT is useful for writing design reports, which can help improve report writing skills. However, it is not useful for reflecting on the design and design process, which requires students' own thoughts and experiences. Overall, ChatGPT seems to be a useful tool in assisting students in various aspects of the engineering design process, but it is not a substitute for students' own creativity and critical thinking skills.
In summary, the professor believes that ChatGPT is useful for writing design reports, which can help improve report writing skills. However, it is not useful for reflecting on the design and design process, which requires students' own thoughts and experiences. Overall, ChatGPT seems to be a useful tool in helping students in certain aspects Engineering Design, but it is not a substitute for students' own creativity and critical thinking skills.


====Learning goals====
====Learning goals====
Upon examining the outcomes of the survey, student ChatGPT expert interviews, and university professor interview concerning the usage of ChatGPT in achieving the learning objectives of the Engineering Design course, the following observations were made.  
Based on the mass survey, student ChatGPT expert interviews, and university professor interview regarding the usage of ChatGPT in achieving the learning objectives of the Engineering Design course, the following observations were made.  


During the ChatGPT expert interviews, the interviewees give conflicting signals over the approach to the learning goals of the course with ChatGPT. Two extremities can be found even, where one student would use ChatGPT for almost every step of the process and the other notes that ChatGPT lacks information of the course and group progress that a student may have in their head and thus cannot help during these processes. There does not seem to be overwhelmingly common answers between the experts.  
During the ChatGPT expert interviews, the interviewees give conflicting answer about the achievability of the learning goals of Engineering Design with ChatGPT. Two extremities can be found, where one student would use ChatGPT for almost every objective and the other for none, as they note that ChatGPT lacks information of the course and group progress that a student may have in their head and thus cannot help during these processes. There does not seem to be overwhelmingly common answers between the experts.  


The survey results show there are always at least 30% of students that think a learning objective can be reached with ChatGPT and at least 23% that think they cannot.  
The survey results show there are always at least 30% of students that think a learning objective can be reached with ChatGPT and at least 23% that think they cannot.  
Line 720: Line 717:
Table 6 shows the combined opinion of the five ChatGPT experts and professor regarding the extent to which ChatGPT can help achieve certain learning objectives.  
Table 6 shows the combined opinion of the five ChatGPT experts and professor regarding the extent to which ChatGPT can help achieve certain learning objectives.  


The professor and experts have some similarities in their views on the feasibility of the learning objectives using ChatGPT. The table reveals that both the ChatGPT experts and the professor agreed that ChatGPT cannot help achieve the learning objective of "Making a final design choice between a number of concepts." This indicates a clear limitation of ChatGPT's usefulness in this particular aspect of the engineering design process.
The professor and experts have some similarities in their views on the achievability of the learning objectives using ChatGPT. The table reveals that both the ChatGPT experts and the professor agreed that ChatGPT cannot help achieve the learning objective of "Making a final design choice between a number of concepts", as this should be entirely done by the design group itself. For the learning objectives "Develop a detailed design that meets the specifications," and "Define the functional and technical specifications," both the ChatGPT experts and the professor agreed that ChatGPT can help achieve those objectives. However, for the remaining learning objectives, the opinions were divided or inconclusive.  


For the learning objectives of "Execute a generic design process," "Develop a detailed design that meets the specifications," and "Define the functional and technical specifications," both the ChatGPT experts and the professor agreed that ChatGPT can help achieve those objectives.
Overall, it seems that the professor is more cautious about using ChatGPT for creative aspects of the course, such as formulating a design goal and generating realization possibilities. The students, on the other hand, seem to have more varied opinions on the achievability of these objectives with ChatGPT. The differing opinions and inconclusive results indicate the need for further research and evaluation to better understand the effectiveness of ChatGPT in helping students achieve the learning objectives of Engineering Design.
===Limitations===
In this research, there are multiple points for discussion regarding the results and their interpretation.


However, for the remaining learning objectives, the opinions were divided or inconclusive. Some of the objectives received mixed responses from the experts and the professor, such as "Generating an elaborate list of realization possibilities for the different functions of the design." In other cases, the responses were inconclusive, such as "Developing and executing a test plan for the prototype" and "Evaluating a prototype based on test results and giving advice for redesign."
First and foremost, the study conducted was relatively small compared to what it was trying to prove. To prove the hypothesis that certain learning goals would be affected by the use of ChatGPT, it would have been necessary to consult with multiple experts in the AI and educational fields, and talk to more teachers and even tutors of the course. However, in this research, only one teacher of the course was interviewed which gives a limited view of the expectations and opinions of the teachers of the TU/e. Next to that, interviewing one teacher can also give a biased view on the topic, as only the opinion of one particular teacher is taken into account. Ideally, multiple teachers of the course should have been interviewed, and potentially alongside tutors of the course s well. Furthermore, to get a clear picture of how the TU/e looks at the use of ChatGPT in courses, an interview with the executive board of the TU/e would have shed light on this. This board is responsible for the management of the school, and thus has an important say in this topic. Furthermore, the academic advisors of the faculties are the contact persons about questions regarding ChatGPT use on the TU/e, so they also could have given valuable insights.  


Overall, it seems that the professor is more cautious about using ChatGPT for creative aspects of the course, such as formulating a design goal and generating realization possibilities. The students, on the other hand, seem to have more varied opinions on the achievability of these objectives with ChatGPT. This also highlights the potential uses and limitations of ChatGPT in the context of Engineering Design. The differing opinions and inconclusive results indicate the need for further research and evaluation to better understand the effectiveness of ChatGPT in assisting with various aspects of the engineering design process.  
Additionally, the response rate to the survey that was sent out was limited, with only 39 students responding. As the survey was meant to provide a good overview of the general student opinion, a higher response rate was hoped for. As the survey targeted the students of the TU/e, and more than 10.000 students study at the TU/e, a sample size of 100 is needed in order to get a 10% margin of error. In this study only 39 responses were registered, which means it will be very hard to conclude anything that is statistically significant out of this data. Ideally we would have reached at least 100 responses, to get a better image of the actual thoughts of the population we were sampling for.
===Limitations===
In this research, there are multiple points of discussion regarding the results and their interpretation. First and foremost, the study conducted was relatively small compared to what it was trying to prove. To prove the hypothesis that certain learning goals would be affected by the use of ChatGPT, it would have been necessary to consult with many experts in the AI and educational fields. However, in this research, only one teacher was interviewed which gives a limited view of the expectations and opinions of the teachers of the TU/e. Next to that interviewing one teacher can also give a biased view on the topic. 


Besides, the few students who were interviewed were not as experienced with ChatGPT as hoped due to the limited time ChatGPT has been available to the public. It was also not stated clearly enough what was meant by ‘experienced with ChatGPT’ which resulted in a big variation in experience among the students that were interviewed. It is important that students have the same base level of expertise as students who are more experienced with ChatGPT tend to think of it as a more useful technology.
Another important point to notice is that the few students who were interviewed were not as experienced with ChatGPT as wanted. This is however due to the limited time ChatGPT has been available to the public, and was out of our control. As ChatGPT is still very new, we can say that there are currently no expert users, only novice users. A lot of people are still trying to get a better understanding of ChatGPT and are still in the process of forming their opinion. This makes it difficult to get valuable insights from these interviews, as the users are just starting out with ChatGPT. Furthermore, it was not clearly defined what was meant by ‘experienced with ChatGPT’ which resulted in variations in experience among the students that were interviewed. Some of the students that were interviewed had used it more, and all students use it to achieve different goals. This makes it difficult to compare the interviews.


Additionally, the response rate to the survey that was sent out was limited, with only 39 students responding. As the survey was meant to provide a good overview of the general student opinion, a higher response rate was hoped for.  
These points lead to the conclusion that the results might be inadequate to confirm or reject the hypothesis.


Another discussion point as earlier named is the limited time ChatGPT is available, therefore it was hard to find students and teachers who were very experienced with ChatGPT. A lot of people are still trying to get a better understanding of ChatGPT and are still in the process of forming their opinion. These four discussion points lead to the conclusion that the results might be inadequate to confirm or reject the hypothesis.
===Future research===
In the future, it would be wise to extend this study to more courses, not just the Engineering Design course. ChatGPT holds the ability to change all courses taught at the TU/e, and other other education facilities. For the TU/e specifically, the courses Calculus, Physics and Data Analysis would be a great start, as these are all courses all students of the TU/e have to follow, and courses that have the potential to be greatly affected by ChatGPT.  


===Future research===
If this study would be conducted again for other courses, it may be necessary to use a more targeted recruitment strategy to gain more responses for the survey. Relying on snowball sampling and social circles of researchers is not enough. Using a reward, like offering money, would be a better way to get more responses.
In future research, it is important to conduct more interviews with experts on the topic of ChatGPT and education.. By conducting more expert interviews, researchers can gain a deeper understanding of the challenges and opportunities associated with implementing chatbots like chatGPT in education. More interviews will also decrease the chance of having biased answers of teachers in the same research groups.  
 
To broaden the scope, research should be conducted into where and how ChatGPT may be used in courses, so that it is clear at the beginning of the course. Courses should be analyzed separately, to investigate for each specific course how ChatGPT can be implemented. However research into the setup of general rules for academic facilities is important to conduct as well. This should achieve transparency about the awareness of ChatGPT as well as its features and limitations.  


In addition, to increase the response rate from students in the survey, it may be necessary to use a more targeted recruitment strategy. For example, researchers could target students who have previously used ChatGPT in a course or who have a strong interest in the use of technology in education. By doing so, researchers can ensure that the sample is representative of the population of interest and have a better chance on a high response rate.
Moreover, it might be wise to wait for more people to gain experience with ChatGPT before conducting further research. As ChatGPT continues to evolve and becomes more widely used, it is likely that more people will become familiar with it. Therefore more studies will be conducted on its potential impact on education. Waiting for more people to gain experience with ChatGPT before conducting further research will give researchers more data to work with what will lead to a more trustworthy conclusion.


It is also important to check whether the students interviewed have the same base level of expertise in the use of ChatGPT. It was previously concluded that to use ChatGPT efficiently it is important to have some skill with it. If some students have more experience or knowledge with ChatGPT than others, it could introduce bias and undermine the validity of the results. Therefore, researchers should consider assessing participants' prior knowledge and experience with ChatGPT and adjust the analysis accordingly.
In future research, researchers could consider conducting a longitudinal study to investigate the long-term impact of ChatGPT on the learning outcomes of students. The researcher would acquire the data over a long period of time. This data would include their learning outcomes which gives researchers the opportunity to analyze the long-term results of ChatGPT on educational learning goals. They could even compare the results of students who did use ChatGPT with the results of students who did not use it.


Moreover, it might be wise to wait for more people to gain experience with ChatGPT before conducting further research. As ChatGPT continues to evolve and becomes more widely used, it is likely that more people will become familiar with it. Therefore more studies will be conducted on its potential impact on education. Waiting for more people to gain experience with chatGPT before conducting further research will give researchers more data to work with what will lead to a more trustworthy conclusion.  
In this research, it became known that there is great need for AI detection tools. These tools are really useful for teachers, so they can check whether a student has written something by themselves or not. If it is not allowed to use ChatGPT, AI detection tools are a great way to check for this. More research has to be done into these tools, to see how reliable they are, and how they can be approved upon. Another important topic mentioned multiple times in our research exam regulations for courses need to be revised. With ChatGPT the possibility to ‘cheat’ on tests becomes more glaring, and rules and regulations need to be set into how and if ChatGPT can be used in exams. However this will also be course specific, and further research is necessary to set these rules and regulations.


Finally, in future research, researchers could consider conducting a longitudinal study to investigate the long-term impact of ChatGPT on the learning outcomes of students. The researcher would acquire the data over a long period of time. This data would include their learning outcomes which gives researcher the opportunity to analyze the long-term results of chatGPT on educational learning goals. They could even compare the results of students who did use chatGPT with the results of students who did not use it.  
Finally, as lecturers educate the experts of tomorrow, there is a need to determine how we can teach students the necessary skills for sensible use of ChatGPT. It is therefore obvious that we need to address the question of how we will integrate such AI tools into higher education, and how students will be taught in the use of ChatGPT. Sensible use needs to be taught in classes, but further research is needed to determine what the best way to do this will be.


Overall, future research should aim to use a variety of methods to obtain a better and more in-depth understanding of the potential impact of chatGPT on education and the learning goals of courses. By doing so, researchers can provide recommendations that are based on actual results and experiences with chatGPT.  
Overall, future research should aim to use a variety of methods to obtain a better and more in-depth understanding of the potential impact of ChatGPT on education and the learning goals of courses. By doing so, researchers can provide recommendations that are based on actual results and experiences with ChatGPT.


==Ethics of Chatbots and Plagiarism==
==Ethics of Chatbots and Plagiarism==
Line 754: Line 753:
The first concern when using ChatGPT in education is privacy. ChatGPT has the ability to talk with students in a human way, which can create a false sense of security and trust. As a result, there is a risk that students may share sensitive information about their personal lives with ChatGPT. To decrease this risk, educators should take steps to establish clear guidelines and warnings around the use of ChatGPT in the classroom, emphasizing the importance of safeguarding personal information.
The first concern when using ChatGPT in education is privacy. ChatGPT has the ability to talk with students in a human way, which can create a false sense of security and trust. As a result, there is a risk that students may share sensitive information about their personal lives with ChatGPT. To decrease this risk, educators should take steps to establish clear guidelines and warnings around the use of ChatGPT in the classroom, emphasizing the importance of safeguarding personal information.


Another concern is that ChatGPT could provide false information to the student and the accountability of this. Because ChatGPT learns from its conversations with users, it is crucial that its responses are reviewed to make sure answers are reliable and accurate. If students are not aware of this risk, they may blindly trust ChatGPT's information, leading to them learning and using the wrong information for their schoolwork. In such cases, it can be difficult to determine who is accountable for the spreading of false information. As such, it is essential that educators clearly communicate the limitations of ChatGPT to students and provide guidance on how to evaluate its responses critically.
Another concern is that ChatGPT could provide false information to the student and the accountability of this. Because ChatGPT learns from its conversations with users, it is crucial that its responses are reviewed to make sure answers are reliable and accurate. If students are not aware of this risk, they may blindly trust ChatGPT's information, leading to them to learn and to use the wrong information for their schoolwork. In such cases, it can be difficult to determine who is accountable for the spreading of false information. As such, it is essential that educators clearly communicate the limitations of ChatGPT to students and provide guidance on how to critically evaluate its responses.


Lastly, the issue of plagiarism arises with the use of ChatGPT in education. In this research, the students who were interviewed were asked whether they thought using ChatGPT for their schoolwork should be considered plagiarism. The general opinion came down to that if you let it complete a whole task for you, like writing a report, it should be considered plagiarism. If you use it as a way to produce a whole essay or parts of an essay it should be reported, otherwise it should be considered plagiarism. Just as if someone doesn’t report a source used for a report. However, if you’re using it as a tool to help you it is not considered plagiarism. The teacher that was interviewed agreed with the fact that if ChatGPT is used as a tool to help it should not be considered plagiarism. Nevertheless, it is important for educators to clearly state whether it is allowed to use ChatGPT for an exercise or course.  
Lastly, the issue of plagiarism arises with the use of ChatGPT in education. In this research, the students who were interviewed were asked whether they thought using ChatGPT for their schoolwork should be considered as plagiarism. The general opinion came down to that if you let it complete a whole task for you, like writing a report, it should be considered plagiarism. If you use it as a way to produce a whole essay or parts of an essay, it should be properly cited, otherwise it should be considered plagiarism. Just like when someone does not report a source used for a report. However, if you’re using it as a tool to help you, it is not considered plagiarism. The teacher that was interviewed agreed with this statement. Nevertheless, it is important for educators to clearly state whether it is allowed to use ChatGPT for an exercise or course.  


In conclusion, while ChatGPT has the potential to make education more efficient, it is very important that educators take a proactive approach to addressing the ethical issues that come along with its use. By being mindful of the risks related to privacy, accountability, and plagiarism, and by providing students with clear guidance and expectations, educators can ensure that the use of ChatGPT in education is both safe and effective.
In conclusion, while ChatGPT has the potential to make education more efficient, it is very important that educators take a proactive approach to addressing the ethical issues that come along with its use. By being mindful of the risks related to privacy, accountability, and plagiarism, and by providing students with clear guidance and expectations, educators can ensure that the use of ChatGPT in education is both safe and effective.


==Conclusion==
==Conclusion==
It is difficult to come to a clear conclusion about the hypothesis "Most of the learning goals of the course engineering design could be affected by the use of ChatGPT, therefore the learning goals of the course might not be achieved by the students who use ChatGPT."
In the presented study, we investigated the influence of ChatGPT on students in reaching the learning objectives of the course Engineering Design at the TU/e.


While some students believe that ChatGPT can be useful in achieving certain learning objectives, others have expressed concerns about its limitations, particularly in creative thinking and problem-solving. Additionally, the effectiveness of ChatGPT appears to be highly dependent on the user's skill level and familiarity with the learning objectives of the course.
This report discussed the implications of using ChatGPT in Engineering Education. While these large language models like ChatGPT have the potential to revolutionize the way we access and use information, they have also raised concerns about academic integrity, student engagement, and the role of teachers in the learning process. Some of the issues that arise include the potential for cheating, the unclear definition of authorship, and the possibility of plagiarism. However, the use of AI models can also enhance student engagement, collaboration, and accessibility.  


The interview with the university professor suggests that ChatGPT can be a valuable tool for certain tasks, such as improving report writing and searching for literature, but may not be sufficient for more complex or creative learning objectives.
A mixed-methods approach, including a survey and interviews were used to understand the opinions and experiences of participants with ChatGPT in the context of Engineering Design. The survey was sent to university students who had completed the course, and the interviews were conducted with a professor from the Engineering Design department and frequent users of ChatGPT. The study aimed to gain insights into the frequency of ChatGPT usage, its perceived usefulness, and its impact on achieving the learning objectives of Engineering Design. The survey and interviews together aimed to provide a comprehensive understanding of the participants' opinions on the use of ChatGPT in the Engineering Design course.  


Overall, it seems that the use of ChatGPT in the course Engineering Design could have both positive and negative effects on achieving the learning goals, and its effectiveness will depend on the specific objectives and the skill level of the students using it. Further research may be needed to fully assess the impact of ChatGPT on the course's learning outcomes.
The survey found that students primarily used ChatGPT for generating text in response to prompts, answering questions, and assisting with writing tasks such as essays and code. The results also suggest that students are divided on whether ChatGPT should be used in the Engineering Design course, with some seeing its potential benefits while others are hesitant. Despite these limitations and concerns, the survey results suggest that ChatGPT could be a valuable tool to complement students' learning experiences in Engineering Design. Additionally, the effectiveness of ChatGPT appears to be highly dependent on the user's skill level and familiarity with the learning objectives of the course.


*
Based on the interviews, student experts identified gaining information, receiving writing assistance, and saving time as the primary reasons for using ChatGPT. However, the most frequently mentioned disadvantage was its limited functionality, particularly in academic settings where it is believed to only handle straightforward tasks. In regards of the course Engineering Design, the students believe that ChatGPT could be helpful in certain aspects of the design process, such as formulating a design goal and writing a design report, defining functional and technical specifications, and executing a generic design process. However, they do not believe it would be particularly useful for more complex tasks such as selecting design concepts or developing detailed designs.


==Survey==
During the interview with the professor he shared his positive experience with ChatGPT and encourages his students to use it, citing benefits such as improved report writing, efficient search capabilities, and faster completion of tasks. However, he notes that ChatGPT's main limitation is its limited database and lack of creativity, which prevents it from generating innovative ideas. While ChatGPT can help with selecting design concepts and structuring concepts for detailed design, students must do the thinking and actual work themselves. Overall he found ChatGPT a useful tool but it cannot replace students' own creativity and critical thinking skills.
https://forms.office.com/Pages/DesignPageV2.aspx?subpage=design&FormId=R_J9zM5gD0qddXBM9g78ZOciNcKFkhVJsg7I7H6kiF5UNzNXQzhaOEpWSjZJSDg1WUZBN1lWTkkwUS4u


Comparing these results with our hypothesis, which was based on current literature and on what we thought was do-able with ChatGPT current abilities (Table 6), tells us that there is a big difference in experience in how ChatGPT can and is used. The student experts and professor had mixed opinions on the usefulness of ChatGPT, with some finding it helpful for certain tasks and learning objectives, while others found it lacking in information and unable to assist in more creative aspects. These mixed answers left us with an inconclusive result about ChatGPTs effectiveness with achieving the learning goals of Engineering Design.
Overall, it seems that the use of ChatGPT in the course Engineering Design could mainly have positive effect on achieving the learning goals, since almost all the students expressed that they would use ChatGPT as an assisting tool and not as a replacement of their own work. Its effectiveness will depend on the specific objectives and the skill level of the students using it. Further research may be needed to fully assess the impact of ChatGPT on the course learning outcomes. A certainty however is that almost no learning objectives can be completely circumvented by ChatGPT. Nevertheless, there is still a lot of debate and uncertainty about the achievability and the way students will use ChatGPT for the course.
Based on this research, the expectation is that students will try to use ChatGPT for some aspects of the course. However, usually only in a supportive role. ChatGPT is not flawless and students seem to be aware of this. Therefore the advice is to embrace ChatGPT, since it is not used for everything and it can still be a valuable addition to the curriculum of students to enhance their final deliverables. Furthermore, based on the thematic analysis, it’s clear that the professor and experts warn about the pitfalls of using ChatGPT. Therefore, the course could even go as far as to teach its students about the usability and limitations of ChatGPT, so Engineering Design will continue to teach the necessary skills to be expected from a future engineer.
==Bibliography==
==Bibliography==
<references /><br />
<references /><br />
Line 780: Line 783:


===1. Survey questions===
===1. Survey questions===
https://forms.office.com/Pages/DesignPageV2.aspx?subpage=design&FormId=R_J9zM5gD0qddXBM9g78ZOciNcKFkhVJsg7I7H6kiF5UNzNXQzhaOEpWSjZJSDg1WUZBN1lWTkkwUS4u
'''<u>Research into ChatGPT</u>'''
'''<u>Research into ChatGPT</u>'''


Line 991: Line 996:
Before we go into the specific course, we’d like to know what your knowledge and experience is regarding ChatGPT
Before we go into the specific course, we’d like to know what your knowledge and experience is regarding ChatGPT


# What is your knowledge and experience with the usage of ChatGPT?
#What is your knowledge and experience with the usage of ChatGPT?
#* Specifically:
#*Specifically:
#*# Design / creativity
#*#Design / creativity
#*# Coding
#*#Coding
#*# Writing  
#*#Writing
#*# Summarizing / explaining (search engine)
#*#Summarizing / explaining (search engine)
#* How would you describe your overall experience with ChatGPT?
#*How would you describe your overall experience with ChatGPT?




Line 1,019: Line 1,024:
Now we will go more in depth about the potential usage of ChatGPT in academic education, and more specifically, Engineering Design.
Now we will go more in depth about the potential usage of ChatGPT in academic education, and more specifically, Engineering Design.


# What is your opinion on using ChatGPT as an academic tool?  
#What is your opinion on using ChatGPT as an academic tool?
# Does using ChatGPT count as plagiarism in your opinion?
#Does using ChatGPT count as plagiarism in your opinion?


These are the learning objectives of Engineering Design as listed on Canvas:
These are the learning objectives of Engineering Design as listed on Canvas:


* Execute a generic design process
*Execute a generic design process
* Formulate a design goal
*Formulate a design goal
* Define the functional and technical specifications
*Define the functional and technical specifications
* Generate an elaborate list of realization possibilities for the different functions of the design
*Generate an elaborate list of realization possibilities for the different functions of the design
* Select a number of design concepts from an extensive list of realization possibilities
*Select a number of design concepts from an extensive list of realization possibilities
* Make a final design choice between a number of concepts
*Make a final design choice between a number of concepts
* Develop a detailed design that meets the specifications
*Develop a detailed design that meets the specifications
* Develop and execute a test plan for the prototype
*Develop and execute a test plan for the prototype
* Evaluate a prototype based on test results and give an advice for redesign
*Evaluate a prototype based on test results and give an advice for redesign
* Reflect on the design and on the design process
*Reflect on the design and on the design process
* Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design
*Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design


<br />
<br />


# How do you expect the students to reach these learning objectives?
#How do you expect the students to reach these learning objectives?
## What steps do they need to take?
##What steps do they need to take?
## What tools do they need to use / are allowed to be used?
##What tools do they need to use / are allowed to be used?
## How do you think chatbots like ChatGPT would influence students’ ability to obtain the learning objectives?
##How do you think chatbots like ChatGPT would influence students’ ability to obtain the learning objectives?
## What would you change about the learning objectives? (For example, make them more complex)
##What would you change about the learning objectives? (For example, make them more complex)
## Compare his answers to those of students
##Compare his answers to those of students
 
#What effect do you think ChatGPT will have on this course in general?
#In your opinion, does ChatGPT make the education of students better, or does it hinder them in their development?


# What effect do you think ChatGPT will have on this course in general?
# In your opinion, does ChatGPT make the education of students better, or does it hinder them in their development?




'''Survey input'''[[File:Survey learning goals.png|none|thumb|Figure 1. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"]]Regarding Engineering Design (outdated):
'''Survey input'''[[File:Survey learning goals.png|none|thumb|Figure 1. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"]]Regarding Engineering Design (outdated):


* 48% of students think ChatGPT could've had a positive impact on their project
*48% of students think ChatGPT could've had a positive impact on their project
* 58% of students see ChatGPT as a useful tool
*58% of students see ChatGPT as a useful tool
* 54% of students would use ChatGPT for tedious tasks
*54% of students would use ChatGPT for tedious tasks
* Only 32% would use ChatGPT regularly
*Only 32% would use ChatGPT regularly




Line 1,063: Line 1,069:
Some examples of what ChatGPT can do:
Some examples of what ChatGPT can do:


* Write emails
*Write emails
* Write essays and academic reports
*Write essays and academic reports
* Write poetry and song lyrics
*Write poetry and song lyrics
* Compose music
*Compose music
* Answer (test) questions and solve problems
*Answer (test) questions and solve problems
* Generate lines of code based on a prompt
*Generate lines of code based on a prompt
* Answer customer queries
*Answer customer queries




Line 1,076: Line 1,082:
These are the learning objectives of Engineering Design as listed on Canvas:
These are the learning objectives of Engineering Design as listed on Canvas:


* Execute a generic design process
*Execute a generic design process
* Formulate a design goal
*Formulate a design goal
* Define the functional and technical specifications
*Define the functional and technical specifications
* Generate an elaborate list of realization possibilities for the different functions of the design
*Generate an elaborate list of realization possibilities for the different functions of the design
* Select a number of design concepts from an extensive list of realization possibilities
*Select a number of design concepts from an extensive list of realization possibilities
* Make a final design choice between a number of concepts
*Make a final design choice between a number of concepts
* Develop a detailed design that meets the specifications
*Develop a detailed design that meets the specifications
* Develop and execute a test plan for the prototype
*Develop and execute a test plan for the prototype
* Evaluate a prototype based on test results and give an advice for redesign
*Evaluate a prototype based on test results and give an advice for redesign
* Reflect on the design and on the design process
*Reflect on the design and on the design process
* Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design
*Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design


===4. ChatGPT experts on learning objectives===
===4. ChatGPT experts on learning objectives===

Latest revision as of 16:08, 10 April 2023


As written by ChatGPT:

Artificial Intelligence, a wonder of the modern age

A creation made of code, with endless knowledge in its brain

From data analysis to language skills, it's a tool of great worth

Changing the way we live, and opening doors to new growth and mirth


But we must always be mindful, of the impact it may bring

For AI can be used for good or for deceit, and we must choose the right thing

So let us use it wisely, with integrity as our guide

For the future of our world, is shaped by the choices we decide.


Group members

Name Student ID Department
Famke Peek 1459058 Psychology & Technology
Gabriëlle van Heteren 1605305 Biomedical Engineering
My Tran 1620940 Industrial Design
Naud van Rosmalen 1555464 Biomedical Engineering
Niels van Noort 1613928 Biomedical Engineering
Quincy Netteb 1468634 Psychology & Technology

Introduction

ChatGPT can be tracked back to 2018, to when a first version called Generative Pre-trained Transformer, or GPT-1 for short, was released. This was a very exciting development, however it was not very advanced yet, and therefore did not become very popular. In June 2020, OpenAI released an upgraded version of the GPT-1 model, named GPT-3. This model had a lot more parameters, making it the largest and most extensive language model at that time. It became very popular very quickly, due to its power. This model could generate coherent and human-like responses to a lot of questions, something that the world had not seen before. However, it did take quite some computational power to run it, and therefore GPT-3 was not available for everyone to use. In 2021 OpenAI released yet another updated version, called GPT-3.5, specially designed to be more accessible, and be able to run on a wider range of hardware, even including mobile phones.[1] On November 30th of 2022, the newest version was released, called ChatGPT. This model was built upon the GPT-3.5 version, and was fine-tuned. It has taken the world by storm by what it can do.

The function of ChatGPT that it is used the most, is as a chatbot or virtual assistant. Because of the natural language processing capabilities of ChatGPT, it can simulate human-like conversations rather well and provide its users with support and assistance. ChatGPT has other important uses as well, as it can be used for content generation, language translation and text completion. It is a really exciting new technology, as it has opened up possibilities for natural language processing and artificial intelligence (AI). ChatGPT can do a lot, which also raises ethical concerns, such as in general the potential for misuse of the technology, and how it will impact the job market. Potential concerns aside, ChatGPT also brings us benefits, like its use in education, healthcare and communication. It will be exciting to see how ChatGPT and other models will continue to evolve and impact our lives in the upcoming years.[2]

When focusing more on the field of academic education, ChatGPT has been revolutionary and has had a huge impact. Because of its ability to generate human-like text responses for almost every topic imaginable, it has been proven to be an essential tool for education. Certainly for the times we are living in now, just coming out of a global pandemic, ChatGPT has the power to revolutionize the academic field[3]. At the moment there are major problems with education, such as lack of practical teaching experience, outmoded curriculums, lack of independent learning and lack of innovation in teaching methods[4]. These problems can be combated with implementing AI such as ChatGPT in education.

One of the significant ways that ChatGPT has influenced education is by providing a personalized learning experience[5][6]. The model has the capacity to analyse student data and offer tailored learning materials based on the individual's needs, interests, and preferences. This feature is especially important in today's educational landscape, where traditional one-size-fits-all teaching methods are becoming increasingly outdated. By providing personalized learning experiences, ChatGPT has the potential to improve learning outcomes and increase student engagement and motivation. In addition, AI can also help streamline administrative tasks, such as grading and record-keeping, freeing up teachers to focus on instruction and interaction with students.[7][8] Teachers will become more efficient, which could in result increase the quality of education.[9][10] It is however important that teachers get trained in ChatGPT and feel comfortable with using it, for if the teacher is uncomfortable with the AI, the students are less likely to adopt it.[11] So it is important to help teachers to develop necessary digital competencies and skills for using ChatGPT in ethical and informed ways to enhance the student learning experience and attainment of learning outcomes.[12]

Another way that ChatGPT has impacted education is through its language translation capabilities. With ChatGPT, language barriers are no longer a significant obstacle to learning. The model can translate text and speech in real-time, making it possible for students and educators to communicate effectively across different languages and cultures. This feature has opened up new possibilities for global collaboration and knowledge-sharing, and it has also made education more accessible to students from diverse backgrounds. Moreover, ChatGPT has become an essential tool for chatbots and virtual assistants used in education. Chatbots and virtual assistants have become increasingly popular in educational settings, as they can provide 24/7 support to students and answer their questions promptly. For example, new developments in AI can solve 81% of university-level mathematics questions and even come up with new questions.[13] With ChatGPT's ability to hold coherent conversations, these assistants can simulate human interactions and provide personalized support to students, thereby enhancing their learning experience.[14]

However, it is important to note that the implementation of AI in education is still in its early stages and there are many challenges that must be overcome, such as ensuring the privacy and security of student data and addressing ethical considerations and trust issues around the use of AI in the classroom.[15][16][17]It also raises questions of transparency regarding authorship credit and contributions of output by ChatGPT.[18][19] Furthermore, teachers fear that the AI would reduce their role to assistants and they also questioned the accuracy and reliability of the information generated by the system, as ChatGPT was criticized for its lack of understanding of meaning and content and its association with environmental racism and the interests of tech elites.[20][21] The academic texts produced by ChatGPT are often irreversibly produced from ideas of researchers.[22] Even though many teachers are interested in using AI to improve their teaching practice, they also have concerns about the potential impact of AI on student learning, privacy, and job security.[23][24]
Figure 1. ChatGPT

ChatGPT can be a game-changer in the field of education, providing personalized learning experiences, breaking down language barriers, and enhancing communication between educators and students. As technology continues to evolve, the potential for ChatGPT to transform education is significant, and it will be exciting to see how educators and students will leverage this model to create new opportunities for learning and knowledge-sharing. AI will help teachers in content delivery and other instructions, but in the future real life, human teachers might become obsolete.[25] But, while AI has the potential to greatly impact and improve education, it is important to approach its implementation with caution and careful consideration of its limitations and potential risks. This and the outlook and trust of teachers and students in academic education on AI in the classroom is what will be researched. There will be focussed specifically on the course Engineering Design, a first year course mandatory to all TU/e students, where you go through a complete design process, from the first idea to the realization of the product. With the information obtained from a survey and multiple interviews, a deliverable useful for the course coordinators of engineering design will be created, on how to handle ChatGPT in their course.

State of the Art

What is ChatGPT?

ChatGPT is a new technology, but has been extensively researched. ChatGPT is an acronym for Chat Generative Pre-trained Transformer and was created in November 2022. It is based on the OpenAI GPT-3 engine and has been fine-tuned by supervised and reinforcement learning technology; this means that the AI learns by having humans simulate artificial conversations with it and adapting its responses based on how accurately they reflect natural human dialogue. ChatGPT is also able to remember previously given prompts in the same conversation, making it a more personalized chatbot compared to its alternatives.[1] ChatGPT has a lot of functionalities, for example:

  • Write and debug code, generate scripts and functions
  • Give detailed explanations on complex topics (answer test questions)
  • Write texts in different styles (write student essays)
  • Compose music
  • Explain mathematical theorems
  • Play games like tic-tac-toe

However, the technology is not without its limitations. It has the potential for over-optimization due to its reliance on human oversight, also known as Goodhart's law[2], which could hinder performance. Furthermore, language models like ChatGPT are prone to writing plausible-sounding but incorrect answers, which is called artificial intelligence hallucination[7]; this can be attributed to insufficient training data. The AI is also limited by a lack of knowledge about events that occurred after 2021 and in some cases suffers from algorithmic biases. Furthermore, although ChatGPT is able to produce results that seem genuine, it is unable to fully comprehend the complexity of human language and instead relies solely on statistical knowledge and patterns.

Research into ChatGPT still has a long way to go. Van Dis, E. et al. have proposed five priorities for future research on ChatGPT:

  • The first priority is to explore the ethical implications of AI-generated content and to develop guidelines for responsible use.
  • The second priority is to investigate the limitations of ChatGPT and to develop methods for detecting and addressing bias in its outputs.
  • The third priority is to improve the interpretability of ChatGPT and other NLP models, making it easier to understand how they arrive at their outputs.
  • The fourth priority is to investigate the potential of ChatGPT in domains beyond language, such as image and video analysis.
  • Finally, the fifth priority is to develop more efficient and sustainable methods for training and deploying ChatGPT, in order to reduce its carbon footprint and energy consumption.

In order for research into ChatGPT to advance however, collaboration across disciplines on ChatGPT and other AI systems is very important.[14]

ChatGPT in academic education

Artificial intelligence has been a rapidly growing field in recent years, and the development of large language models like ChatGPT has been at the forefront of this expansion. These models have the potential to revolutionize the way we access and use information, particularly in educational contexts. However, they have also raised concerns about issues like academic integrity, student engagement, and the role of teachers in the learning process. To address these concerns, researchers have conducted various studies and published papers on the implications of ChatGPT for education.

ChatGPT has the potential for using large language models such as GPT for cheating in online exams. For instance, a recent study demonstrated that a chatbot named ChatGPT tried to answer exam questions by providing it with a large amount of relevant data. The results of the study show that ChatGPT was able to achieve high accuracy in answering exam questions, even when the questions were designed to be difficult and require reasoning skills. The authors suggest that this poses a significant threat to the integrity of online exams and call for further research into developing more secure methods for online assessments. [26][27]

Another issue that comes up regarding academic integrity, is the unclarity on who the rightful author of the texts, ideas and inventions should be. Currently, there is no clear consensus on who should be considered the rightful author of AI-generated content. This is a complex issue that requires input from experts in moral philosophy, law, and computer science.[28]

One more issue regards plagiarism when using ChatGPT. A study into this examined the accuracy and originality of scientific abstracts generated by ChatGPT compared to those produced by human experts. The study used a combination of artificial intelligence output detectors, plagiarism detectors, and blinded human reviewers to analyse the abstracts. The results showed that ChatGPT-generated abstracts had a higher similarity score with original abstracts than with other sources, but also revealed some cases of potential plagiarism. This suggests the need for caution when using AI-generated abstracts in scientific research. In the academic world, artificial intelligence models are gaining popularity due to their ability to enhance student engagement, collaboration, and accessibility. A paper, authored by D. Cotton, explores the potential benefits and challenges of using AI in education. The models provide a platform for asynchronous communication, personalized and interactive assessments, and real-time grading and feedback. However, also according to D. Cotton there are concerns about academic integrity, particularly the possibility of plagiarism. With access to GPT-3, students could submit essays that are not their own work. Additionally, there are concerns about inequities in assessment, as students with access to GPT-3 have an advantage over those who do not. The article suggests solutions to combat these challenges, such as asking students to submit a draft before the final essay, set strict guidelines or monitor student work closely.[29]

Another paper explored the broader implications of ChatGPT for traditional models of education. The author argued that AI systems like ChatGPT have the potential to disrupt the traditional "banking model" of education, in which knowledge is transmitted from teacher to student. Instead, learners could access knowledge and information more directly through AI systems, which could democratize access to education and make it more inclusive. The author emphasizes the need for educators to adapt to this changing landscape and to shift their focus towards fostering critical thinking, creativity, and other skills that cannot be easily automated, and suggests that AI systems like ChatGPT have the potential to democratize access to education, making it more accessible and inclusive for learners around the world.[30]

Furthermore, a paper written by Rudolph also explores the implications of ChatGPT for both students and teachers. While ChatGPT offers personalized AI tutoring for students, it also has the potential to reduce the workload for teachers, especially with the use of automatic assessment tools. Furthermore, it can help teachers analyse their students' skills more easily. The paper gives a general overview of challenges and opportunities:[31]

Table 1. Challenges and opportunities of AI in education
Challenges Opportunities
Teachers are afraid students will outsource all their work to ChatGPT. Opportunity for teachers to improve/change their assessment and teaching techniques.
ChatGPT doesn’t evaluate the relevance of the information, it just generates text that is an imitation of what it has learned. “ChatGPT allowed students to learn through experimentation and experience”

To look further into the fear of teachers that students will outsource all their work to ChatGPT, there will be looked to another paper written by García-Peñalvo that reviews previous literature on ChatGPT. The most controversial issue with ChatGPT is the possibility of students using it as an easy solution to write essays without putting in the necessary effort. Because of this, they won’t acquire the needed knowledge for their course. However, the problem might not be the tool itself, but that the assessment techniques of educational institutions have become outdated However, the paper argues that prohibiting ChatGPT is not the way to go. Instead, teachers and students should learn how to use the tool to their advantage.[32]

As mentioned times before, ChatGPT has it obvious limitations. A controversial view in this paper written by Thorp, the author of the article is not necessarily worried about the use of ChatGPT in education as ‘it did well finding factual answers, but the scholarly writing still has a long way to go’. He thinks it pushes academics to design their courses in such a way that they are not easily solved by AI. The author is more worried about the influence of ChatGPT on the world of literature.[33]

While ChatGPT has its limitations, it has the potential to replace humans in routine tasks such as homework grading, potentially spelling the end of traditional essay writing assignments. "ChatGPT User Experience: Implications for Education" explores the user experience of ChatGPT and its potential impact on education. He also states that using AI tools to perform certain tasks ‘should be a part of the educational goals in the future’. The author concludes that teachers need to change their assignments to make it harder for students to use AI, but also notes that using AI in assignments to engage students in learning is a viable option. [34]

To summarize, ChatGPT is a powerful tool with many functionalities, from writing texts and composing music to solving mathematical problems and playing games. However, it is not without its limitations, such as over-optimization, algorithmic biases, and the potential for incorrect responses due to insufficient training data. The future research priorities include exploring the ethical implications of AI-generated content, detecting and addressing bias in its outputs, improving the interpretability of ChatGPT and investigating the potential of ChatGPT in domains beyond language.

In academic education, ChatGPT has the potential to revolutionize the way we access and use information, but it also raises concerns about academic integrity, student engagement, and the role of teachers in the learning process. Further research is needed to develop secure methods for online assessments and determine the rightful author of AI-generated content. Despite the challenges, AI systems like ChatGPT have the potential to democratize access to education and make it more inclusive.

Problem statement

In the presented study, the influence of ChatGPT on students in reaching certain learning objectives is investigate. Therefore, the problem statement is as follows:

How could the use of ChatGPT affect students in reaching the learning objectives of the course Engineering Design (4WBB0) at the TU/e?

Hypothesis

The course engineering design has 13 learning goals. Each one could be affected by the use of ChatGPT. Considering the literature and the known capabilities of ChatGPT, a hypothesis for each learning goal can be made.

  • Execute a generic design process

ChatGPT would not be able to execute a generic design process, therefore this specific learning goal would not be affected by ChatGPT.

  • Formulate a design goal

As ChatGPT is an AI language model it is capable of formulating a design goal, provided that the users ask the right question. Therefore this learning goal could be affected.

  • Define the functional and technical specifications

As ChatGPT is an AI language model it is capable of defining the functional and technical specifications. However, the user would have to need to provide an analysis of the design problem, the target audience, and the desired outcomes in order to get valuable specifications. This learning goal could be affected.

  • Generate an elaborate list of realization possibilities for the different functions of the design

As ChatGPT is an AI language model it is capable of formulating an elaborate list of realization possibilities for the different functions if the requirements and specifications of the design are given. Therefore this learning goal could be affected.

  • Select a number of design concepts from an extensive list of realization possibilities

As ChatGPT is an AI language model it is capable of making selecting some designs that fit the design goal the best. Therefore this learning goal could be affected.

  • Make a final design choice between a number of concepts

As it is hard to give ChatGPT a complete overview of the context and specifications needed for the design a final design choice would be have to made by the student. Therefore this learning goal is not affected.

  • Develop a detailed design that meets the specifications

As it is hard to give ChatGPT a complete overview of the context and specifications needed for the design a complete detailed design is hard to generate. However, it could give the student a start with the design and only need some adjustments. Therefore this learning goal is not affected

  • Develop and execute a test plan for the prototype

As the ChatGPT is not a physical AI it can’t execute a test plan. Therefore this learning goal will not be affected.

  • Evaluate a prototype based on test results and give an advice for redesign

ChatGPT is capable of this and therefore this learning goal could be affected.  

  • Reflect on the design and on the design process

As it is hard to explain ChatGPT the whole design process a group has been through, reflecting on it would be difficult. Therefore this learning goal would not be affected.

  • Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design

ChatGPT is capable of writing separate parts of the report and this learning goal could therefore be affected.

Table 2. A quick overview of the hypothesis
Can It be done by ChatGPT: Yes No
Execute a generic design process X
Formulate a design goal X
Define the functional and technical specifications X
Generate an elaborate list of realization possibilities for the different functions of the design X
Select a number of design concepts from an extensive list of realization possibilities X
Make a final design choice between a number of concepts X
Develop a detailed design that meets the specifications X
Develop and execute a test plan for the prototype X
Evaluate a prototype based on test results and give an advice for redesign X
Reflect on the design and on the design process X
Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design X

Method

To address the research question of the study, a mixed-methods approach was adopted. This involved the use of both a survey and interviews to obtain a comprehensive understanding of the participants' opinions and experiences with ChatGPT in the context of Engineering Design.

The survey was sent to university students who had completed the Engineering Design course at TU/e, with the aim of gaining insights into their usage and perception of ChatGPT. The survey included questions related to the frequency of ChatGPT usage, perceived usefulness of the tool, and its impact on reaching the learning objectives of Engineering Design.

A panel discussion hosted by the TU/e about ChatGPT in Engineering Education was attended. From this and background literature, a series of interviews were conducted with both a professor from the Engineering Design department and frequent users of ChatGPT: which will be called experts. The interviews aimed to provide a deeper understanding of the participants' opinions and experiences with ChatGPT, specifically in the context of Engineering Design. The interviews with the student experts aimed to capture their motivation for using ChatGPT, how they use the tool, and their perception of its impact on the Engineering Design learning goals. The interview with the professor aimed to capture the impact they believe ChatGPT has on the learning objectives, and their overall opinion on the use of ChatGPT.

Together, the survey and interviews aimed to obtain a comprehensive understanding of the participants' opinions regarding the use of ChatGPT in the Engineering Design course and its impact on achieving the learning objectives within this course.

Survey

The survey consists of questions about the participants' general knowledge and experience about ChatGPT, and how this tool could affect them reaching the learning goals of Engineering Design. The questions from the survey can be found in appendix 2.

Participants

The survey was distributed to students who are enrolled at the TU/e and who have finished the course Engineering Design, which are 2nd and higher year students. The participants were obtained by asking students in the social circles of the researchers, and spreading a link in text group chats. There were a total of 39 responses.

Procedure

The outline of the survey was as follows (see appendix 2): a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals.

Panel discussion

On 23rd of march, the TU/e hosted a panel discussion about ChatGPT in Engineering Education. This panel included an associate professor in the department of philosophy and ethics who is doing a study on disruptive technologies, an assisting professor in the department of philosophy and ethics who is doing a study on the philosophy of AI, a Data Science master student, a professor in Computer Science, a Complied Physics master student, a professor in the department of Mechanical Engineering, the program manager of Education Innovation and Technology who is also an advisor for the examination committee here on the TU/e and the host CTO of FruitPunch AI and former Computer Science student here at the TU/e. This discussion was observed and notes were taken of the things that were interesting to use when conducting the interviews later on.

Interview with ChatGPT experts

Student experts who frequently use ChatGPT were interviewed to gain a better understanding of how and why they use ChatGPT, and how this usage could impact achieving the learning goals of Engineering Design.

Participants

The focus was on university students who were all enrolled at the TU/e and had successfully completed the Engineering Design course. A total of five participants were recruited, one female and four males, ranging from the ages 20 to 23 (Mage= 21, SD = 1.79, 20% female). The participants were sampled by the convenience sampling method, all the participants had to be students at the University of Eindhoven, and they should have taken the course Engineering Design, since this was a crucial part for our research.

Procedure

The outline of the interview was as follows: a brief section about their general knowledge of ChatGPT, their opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participants, and the latter to measure the effects of ChatGPT on achieving certain learning goals. Open questions were asked that were mentioned in the interview script, see appendix 3. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analysed carefully, which gave a certain conclusion of the interview.

Interview with university professor

In addition, an interview was conducted with an associate professor from TU/e who teaches the course Engineering Design. This interview aimed to provide insights into the perspectives of educators regarding the use of ChatGPT and its impact on learning outcomes, particularly in the context of Engineering Design.

Participants
The participant for this interview was Joris Remmers, an associate professor of Mechanical Engineering, whose role within Engineering Design is to organize assessments and set up the organizational structure of it. He is responsible for the assessment setting and rubrics.
Figure 2. Joris Remmers

Procedure

The outline of the interview was as follows: a brief section about his general knowledge of ChatGPT, his opinions and experience with using the chatbot as an academic tool, and finally, a more in-depth section with questions about hypotheticals of using ChatGPT within the course Engineering Design and what the effects of it would be on achieving all the learning goals. The first two sections were to measure the proficiency of the participant, and the latter to measure the effects of ChatGPT on achieving certain learning goals. Open questionswere asked that were mentioned in the interview script, see appendix 4. If the participant did not give a clear answer, follow-up questions were asked to get more in-depth answers. This was followed by transcribing the interview and discussing the interview by adding codes to the transcript. Afterward, similar codes were placed in certain themes. These themes were then analysed carefully, which gave a certain conclusion of the interview.

Thematic analysis

All the interviews have been analysed with the use of thematic analysis. Thematic analysis makes use of finding patterns in the answers to the interview questions. The thematic analysis was conducted via Dedoose[35]. Codes for themes in the interview were made based on the apparent feelings the interviewees have about ChatGPT and its use. Three main groups were established: Features, what interviewees use it for and why they use it; feelings, what the general feeling of interviewees is about ChatGPT; and warnings, what interviewees think are drawbacks of ChatGPT. These results were used to more easily interpret and compare the answers given in the interviews.

Results

User proficiency

Figure 3. Learning goal obtainability by user proficiency

To get a clearer image of the results of the interviews, it is wise to compare the difference in answers between students who were very proficient in using ChatGPT an students who used it now and then for more superficial use. As can be seen in the jitter plot above (figure 3) there is no clear correlation to be seen in the data.

Survey analysis

In the survey we started with some general questions. All participants have heard of ChatGPT before, however 13% do not know its capabilities. When asked to name some capabilities of ChatGPT, students mention the generation of text in response to prompts given, they mention that it gives answers to questions, it helps with writing essays, and writing code. The emphasis is mostly on the fact that you can do a lot with ChatGPT, you can ask it basically anything. Quite a few students in the survey have not used ChatGPT before, specifically 33% of the participants. Most students have used ChatGPT either just once or twice, or weekly. Two participants use it daily, and four participants use it monthly.

In the survey we have asked students what they have used ChatGPT for. The results are very varied: students use it for a lot of different things. Things that students use it for most frequently are just fooling around with it, writing parts of papers, or general academic work. Students also use it for writing emails and summaries, and for writing code or finding errors and solutions in already written code. Several students also mention that they use ChatGPT for answering questions and queries in general, using it more as a search engine. Finally students also mention using it for getting design ideas, and other creative ideas. For an overview look at table 3.

Table 3. Overview of how many times students have mentioned certain ChatGPT functions in the survey
ChatGPT use in general ChatGPT use for academic purposes Can it be used for the course Engineering Design?
Fool around 9x - -
Get design ideas 4x 3x 25x
Other new ideas (creativity) 3x 1x 2x
Answer questions / Explain topics 6x 4x 8x
Writing parts for a paper / academic work 10x - 11x
Writing emails / summaries 8x 3x 3x
Programming help 7x 7x 10x
Problem identification / finding solutions - - 9x

Students were asked to rate the experience with ChatGPT, and on average, the students rated ChatGPT 4 out of 5 stars. We also asked them to rate it as an academic tool, and they rated it 3.5 stars out of 5. These are both pretty high ratings, and the students seem to be happy with ChatGPT and its performance. 65% of students mention that they have not used ChatGPT for academic purposes, however 35% have used it, and for various different courses.

Participants were also asked how they felt about the ownership of output created by ChatGPT. Only 31% felt that usage of ChatGPT should be considered their own work. When asked when the use of ChatGPT should be considered plagiarism, students mention that when it is copied directly from ChatGPT, when it is presented as their own work, and when it does entire exercises for you, they think it should be considered as plagiarism.

In the survey questions specifically relating to the course Engineering Design were asked. 64% of the participants feel that ChatGPT could be a good addition to the course. When asked how they think it can help, students mention getting design ideas quite frequently. Furthermore they also mention getting help with electronics, ChatGPT being a ‘better Google’, help with writing the report and help with coding. Students do mention that they believe for the course creative thinking is required, which they feel ChatGPT is not capable of. Students are divided on whether they think ChatGPT should be used in the course, some say it would be a great addition, others fear it will take away from the learning process or they do not think ChatGPT will add anything useful.

In the survey we provided several statements, and asked the participants their feelings about these.

The first statement: ‘I see ChatGPT as having/could have had a positive influence on my engineering design project.’, most students tend to agree with (33%). However, students are divided on this topic, as 20.5% disagrees. This divide was present in the previous questions as well, where it could be seen students have differing opinions regarding ChatGPT.

The second statement: ‘I would have seen ChatGPT as a useful tool for my engineering design project.’, even more students agree with, namely 41%. There is also a significant percentage of students who are undecided on this matter, namely 25%.

The third statement: ‘I see ChatGPT as a shortcut for tedious tasks I would have/had to perform in the course engineering design.’, the majority of students tend to agree with, 33% of them, and 20% even say they strongly agree. However, on the opposite side, 18% also say they disagree. Most students do see ChatGPT as a shortcut, for helping them with tasks that would have otherwise cost them much more time.

For the fourth statement: ‘I will use/would have used ChatGPT in the course engineering design regularly.’, students very often disagree with. 28% disagrees with this statement, and 28% even strongly disagrees. Only 23% of students agree with this statement. Here it shows as although most students do seem to see some usefulness in ChatGPT, they do not think they will use it that often. It is most likely not something they will integrate into their toolbox and use as a first inclination.

For the final statement: ‘I feel like I would have learned something when using ChatGPT for the engineering design project.’, a large percentage (25%) of students are undecided on this. Students do lean more towards disagreeing with this statement, as 25% disagrees, and only 20% agrees. Here students are really unsure. This comes back to the fact that many students are afraid they will not learn certain skills, because these can now be done with ChatGPT. However, students also seem to realize that these skills might become obsolete, and therefore not as important to master.

Overall students seem to have mixed feelings about ChatGPT and the use of ChatGPT for the course engineering design. Students seem to either see the use in it, and think it is a good addition, or they do not see the use in it, and do not see themselves using it. Students also do not have that much experience with ChatGPT yet, and have not gotten the full grasp of its winnings but also its limitations.

In the survey we also asked the participants to judge the learning goals of the course Engineering Design on how likely they think these goals can be reached with ChatGPT. In figure 4 the answer distribution can be seen.
Figure 4. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"
From this it can be seen that students see ChatGPT being the most useful for formulating a design goal and writing the report. These are both text entries, which could be done with the use of ChatGPT, which students seem to recognize. Furthermore, students also seem to think that defining the functional and technical specifications and executing a generic design process can be done. The former can be directly provided by ChatGPT, but the latter is not something you can ask ChatGPT. However, students do seem to think they can reach this learning goal with the help of ChatGPT. Students seem to think that the learning goal make a final design choice between a number of concepts has the least potential to be done with ChatGPT. They also do not think selecting a number of design concepts from an extensive list of realization possibilities and developing a detailed design that meets the specifications are goals that can be reached with the use of ChatGPT.

Still, students think that quite some learning goals could benefit from or be done with ChatGPT. Here it seems students would think that ChatGPT could be useful for the course.

Panel discussion

On 23rd of march the TU/e hosted a panel discussion about ChatGPT in Engineering Education. A lot of important questions came up; How might it impact workload of teachers? Is the implementation of ChatGPT going to deskill students? How is the TU/e going to check for plagiarism? And is this going to cause data/privacy issues?

During the discussion, the participants shared their thoughts on ChatGPT and its impact on education. They agreed that while it is exciting technology, it can also be disruptive and cause a concern for education. However, they also recognized that human adaptability is strong, and technology like ChatGPT should be given a place. None of the panel members had a clear answer to what this place should be, since it is so hard to check how much a student actually learns when they use ChatGPT.

The participants emphasized the importance of clear communication from TU/e and its teachers about what they teach and their purpose. They also discussed the need to figure out how to view technology like ChatGPT and involve students in the discussion. This challenge presents an opportunity to consider what students really need to learn for their future.

The participants acknowledged that in the short term, students could be asked to produce more output, but long-term solutions require resourcefulness. Overall, they recognized that ChatGPT is both a challenge and an opportunity to think about education's future and how to adapt to new technologies.

Interview analysis

A very concise summary of the interviews with the five ChatGPT experts and professor can be seen in tables 4 and 5 below: these tables depict whether these participants think it would be likely to reach the mentioned learning goal with the use of ChatGPT.

We consulted five student experts and gathered their answers in the appendix table 8, which we used to create table 4. This table shows the answers that most experts agreed on. We considered an answer to be common if at least four out of five experts agreed on it, which corresponds to 80% agreement. For example, if four or more experts answered 'yes' to a question, we assigned 'yes' as the common answer for that learning objective. However, if the experts' answers were too different, meaning that less than four experts agreed, we marked the result as 'inconclusive'.

The results of the expert interviews and professor interview are compared in table 6.

Table 4. ChatGPT experts on whether learning objectives can be achieved with ChatGPT
Learning objective Yes No Inconclusive Y/N (%)
Execute a generic design process X 20/80
Formulate a design goal X 80/20
Define the functional and technical specifications X 80/20
Generate an elaborate list of realization possibilities for the different functions of the design X 60/40
Select a number of design concepts from an extensive list of realization possibilities X 40/60
Make a final design choice between a number of concepts X 20/80
Develop a detailed design that meets the specifications X 80/20
Develop and execute a test plan for the prototype X 40/60
Evaluate a prototype based on test results and give an advice for redesign X 40/60
Reflect on the design and on the design process X 40/60
Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design X 60/40
Table 5. Engineering Design professor on whether learning objectives can be achieved with ChatGPT
Learning objective Yes No Explanation
Execute a generic design process X Students still need to do all the steps themselves, but ChatGPT can be of assistance
Formulate a design goal X This is a creative part and requires out-of-the-box thinking
Define the functional and technical specifications X This is objective knowledge that can be looked up
Generate an elaborate list of realization possibilities for the different functions of the design X This is the most creative part - ChatGPT is not innovative enough
Select a number of design concepts from an extensive list of realization possibilities X Text-book Google work, so ChatGPT can do it, since it can follow strict procedures
Make a final design choice between a number of concepts X ChatGPT can be used to structure the concepts, but students ultimately need to reflect on their own preferences
Develop a detailed design that meets the specifications X This part requires thinking, combining results, solving equations and doing the actual work; ChatGPT can help with parts of it, but not entirely
Develop and execute a test plan for the prototype X Can be done by ChatGPT
Evaluate a prototype based on test results and give an advice for redesign X ChatGPT can compare data easily, especially if it's about software design
Reflect on the design and on the design process X This requires your own thoughts and experiences
Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design X It's wonderful for (improving) report writing
Table 6. ChatGPT experts and Engineering Design professor on whether learning objectives can be achieved with ChatGPT
Learning objective ChatGPT expert Professor Agreement Hypothesis
Execute a generic design process No Yes No
Formulate a design goal Yes No Yes
Define the functional and technical specifications Yes Yes Yes
Generate an elaborate list of realization possibilities for the different functions of the design Inconclusive No Yes
Select a number of design concepts from an extensive list of realization possibilities Inconclusive Yes Yes
Make a final design choice between a number of concepts No No No
Develop a detailed design that meets the specifications Yes Yes No
Develop and execute a test plan for the prototype Inconclusive Yes No
Evaluate a prototype based on test results and give an advice for redesign Inconclusive Yes Yes
Reflect on the design and on the design process Inconclusive No No
Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design Inconclusive Yes Yes

Thematic analysis

In table 7 'Codes for the themes' all the found codes and there descriptions are given.

Table 7. Codes for the themes
Overarching theme Code Description
- Academic usage A mentioning of how ChatGPT can be used in academic circumstances.
- Improved capabilities The notion that ChatGPT is better at doing a certain job than conventional technology or humans.
- Inevitable use When someone mentioned ChatGPT would be used now or in the future, regardless of actions that would be taken against it.
- New learning goal When a new possible learning goal was metioned that arose due to ChatGPT.
- Pluripotent When a interviewee mentioned that they think ChatGPT can do everything or that it has no boundries.
Features Coding Help When ChatGPT was used to write code, help with understanding code, debugging code or optimizing code.
Features Creativity When ChatGPT got used to conduct a creative task where it has to recombine ideas into a new one like coming up with new ideas for a brainstorm or write a caption for a picture.
Features Ease of use When it is mentioned that ChatGPT was easy to use.
Features Fun When ChatGPT was used to mess around, so there was no goal for using it except for momentairy happiness.
Features Providing information When ChatGPT was used to gain information.
Features -Guidance When ChatGPT was used to gain information on how to do a certain task.
Features Summarizing When ChatGPT was used to create a summary af a piece of text or a complex subject.
Features Time saving When the reason for using ChatGPT was given to be saving time compared to other methods of doing the same task.
Features Writing help When ChatGPT was used to help write, for example rewrite a text, improve its grammar or flat out writing it all.
Feelings Neutral The general opinion of about ChatGPT was given to be neutral, the interviewee was not possitive nor negative about the use of it.
Feelings Possitive outlook When the interviewee mentioned they saw a possitive effect of ChatGPT in the future.
Warning Better to do it yourself When the interviewee mentioned they thought a person can do the job better than ChatGPT
Warning Depends on user knowledege The capabilities of ChatGPT depend on how the user uses it. THe more knowledge the suer has, the more ChatGPT can do. In the same way, it is not as usefull in the hands of a inexperienced user.
Warning Hinderance to learning When it was mentioned that current learning goals would be unable to be achieved, since CahtGPT would do the work.
Warning Steep learning curve It is hard for new users to properly learn ChatGPT at the start, a lot of learning needs to be done before ChatGPT can be used prpperly.
Warning Limited When it was noted that ChatGPT could not do something.
Warning Unpersonal When it was mentioned that ChatGPT created a gap between the user and its goal.
These codes are applied to the interviews when their description matches what is mentioned. What follows from it is thematic analysis code occurrence table. Below (figure 5) you can find an overview of the thematic analyses of the interviews.
Figure 5. Thematic analysis code occurrence from the interviews conducted with students who are proficient users of ChatGPT.


The thematically analysed interviews can be found on the following page: https://surfdrive.surf.nl/files/index.php/s/drOP7BwOUh8bzKs

With the password being: 0LAUK0

Discussion

Interpretation

Surveys

Based on the survey results, it becomes clear that university students think that ChatGPT is a very helpful tool that is mainly being used for answering questions, explaining things, and helping with writing essays or code. This shows that students highly value the fact that ChatGPT can be used for a wide range of different tasks. Although many students are familiar with ChatGPT's capabilities, some may not know its full potential.

The overall experience with ChatGPT is considered positive. However, students are not in agreement on whether ChatGPT should be used in the course Engineering Design, with some students seeing the useful possibilities, while others are less enthusiastic. The advocates for the chatbot believe that ChatGPT can be used for certain aspects; specifically, tasks that involve text and writing, such as formulating design goals and writing reports. Additionally, they believe that ChatGPT can help with defining functional and technical specifications, as well as executing a generic design process.

On the other hand, students are less inclined to use ChatGPT for more complex tasks, such as making a final design choice or selecting a number of design concepts from an extensive list of realization possibilities. They also do not believe that ChatGPT can help with developing a detailed design that meets specifications. Another point the students make is about whether using ChatGPT could be considered as plagiarism. Merely a small portion believe that ChatGPT's output should be considered their own original work, while the majority would classify directly copying ChatGPT as plagiarism.

Overall, it seems that students have a realistic understanding of what ChatGPT can and cannot do, and are aware of its limitations. Despite this, they still see potential for its use in certain aspects of Engineering Design, suggesting that it could be a valuable tool to help develop their skills.

Interviews ChatGPT expert

Through thematic analysis, several things can be said regarding the usage of ChatGPT among the interviewed students who are ChatGPT experts. They mentioned that their main reasons for using ChatGPT were gaining information, receiving writing feedback, and to save time, since ChatGPT is a very efficient tool. Other reasons are using it for creativity, academic purposes, and programming help. One interviewee even mentioned that they used ChatGPT as a substitute for Google search.

Interestingly, all interviewees acknowledged the pitfalls of using ChatGPT. The most frequently mentioned disadvantage was its limited functionality, which was a predominant theme. This drawback was especially apparent in academic settings, as interviewees believed that ChatGPT could only handle "straightforward stuff" and that it should be used cautiously. One interviewee even mentioned that ChatGPT "hallucinates" and is not up to date, referring to the fact that ChatGPT's database only reaches up until 2021.

Based on the responses from the student experts who were asked whether they think the learning objectives could be achieved with the help of ChatGPT, it turns out that the experts have differing opinions. Collectively, they believe that ChatGPT has the most potential to help with the learning objectives of formulating a design goal and writing a design report, as these goals focus more on the writing aspect. They also believe that ChatGPT could be helpful in defining functional and technical specifications, and in executing a generic design process. However, the students do not believe that ChatGPT would be particularly useful for selecting a number of design concepts from a list of realization possibilities, developing a detailed design that meets specifications, or evaluating a prototype based on test results and giving advice for redesign.

Overall, the students seem to think that ChatGPT could be a helpful tool for certain aspects of the design process, but it may not be suitable for all tasks. It is important to note that these opinions are based solely on the perspectives of the five interviews students, and may not be representative of all potential users of ChatGPT or all design students.

Interview university professor

The professor, Joris Remmers, was overall positive about his experience with ChatGPT and even encourages his students to use it. The main benefits he mentioned were: improving students' report writing, using it as a more efficient search engine, and helping with speeding up doing ordinary tasks, such as googling for literature. However, according to him, the main drawback of the chatbot is that it is a very linguistic tool. He emphasized the fact that ChatGPT was not creative enough and could not generate innovative ideas due to its limited database.

These statements are backed up by his thoughts on whether it would be likely to reach the learning objectives of Engineering Design with the use of ChatGPT. He believes that ChatGPT can be helpful in some, but definitely not all objectives. ChatGPT can help students in executing a generic design process and defining the functional and technical specifications, as these parts require objective knowledge and can be looked up. ChatGPT can also be used to develop and execute a test plan for the prototype and to evaluate a prototype based on test results, particularly if the prototype is software design.

However, ChatGPT is not suitable for some of the most creative parts of the process, such as formulating a design goal and generating an elaborate list of realization possibilities for the different functions of the design. ChatGPT can help students to select a number of design concepts from an extensive list of realization possibilities by following strict procedures, but students ultimately need to reflect on their own preferences to make a final design choice between a number of concepts. ChatGPT can also be used to structure the concepts for the detailed design, but students need to do the thinking, combining results, solving equations, and doing the actual work.

In summary, the professor believes that ChatGPT is useful for writing design reports, which can help improve report writing skills. However, it is not useful for reflecting on the design and design process, which requires students' own thoughts and experiences. Overall, ChatGPT seems to be a useful tool in helping students in certain aspects Engineering Design, but it is not a substitute for students' own creativity and critical thinking skills.

Learning goals

Based on the mass survey, student ChatGPT expert interviews, and university professor interview regarding the usage of ChatGPT in achieving the learning objectives of the Engineering Design course, the following observations were made.

During the ChatGPT expert interviews, the interviewees give conflicting answer about the achievability of the learning goals of Engineering Design with ChatGPT. Two extremities can be found, where one student would use ChatGPT for almost every objective and the other for none, as they note that ChatGPT lacks information of the course and group progress that a student may have in their head and thus cannot help during these processes. There does not seem to be overwhelmingly common answers between the experts.

The survey results show there are always at least 30% of students that think a learning objective can be reached with ChatGPT and at least 23% that think they cannot.

The professor was also clear in his stance: he believed that ChatGPT would be useful to use for linguistic and more straightforward tasks and learning objectives, but would not be sufficient when it comes to creativity and critical thinking skills.


Table 6 shows the combined opinion of the five ChatGPT experts and professor regarding the extent to which ChatGPT can help achieve certain learning objectives.

The professor and experts have some similarities in their views on the achievability of the learning objectives using ChatGPT. The table reveals that both the ChatGPT experts and the professor agreed that ChatGPT cannot help achieve the learning objective of "Making a final design choice between a number of concepts", as this should be entirely done by the design group itself. For the learning objectives "Develop a detailed design that meets the specifications," and "Define the functional and technical specifications," both the ChatGPT experts and the professor agreed that ChatGPT can help achieve those objectives. However, for the remaining learning objectives, the opinions were divided or inconclusive.

Overall, it seems that the professor is more cautious about using ChatGPT for creative aspects of the course, such as formulating a design goal and generating realization possibilities. The students, on the other hand, seem to have more varied opinions on the achievability of these objectives with ChatGPT. The differing opinions and inconclusive results indicate the need for further research and evaluation to better understand the effectiveness of ChatGPT in helping students achieve the learning objectives of Engineering Design.

Limitations

In this research, there are multiple points for discussion regarding the results and their interpretation.

First and foremost, the study conducted was relatively small compared to what it was trying to prove. To prove the hypothesis that certain learning goals would be affected by the use of ChatGPT, it would have been necessary to consult with multiple experts in the AI and educational fields, and talk to more teachers and even tutors of the course. However, in this research, only one teacher of the course was interviewed which gives a limited view of the expectations and opinions of the teachers of the TU/e. Next to that, interviewing one teacher can also give a biased view on the topic, as only the opinion of one particular teacher is taken into account. Ideally, multiple teachers of the course should have been interviewed, and potentially alongside tutors of the course s well. Furthermore, to get a clear picture of how the TU/e looks at the use of ChatGPT in courses, an interview with the executive board of the TU/e would have shed light on this. This board is responsible for the management of the school, and thus has an important say in this topic. Furthermore, the academic advisors of the faculties are the contact persons about questions regarding ChatGPT use on the TU/e, so they also could have given valuable insights.

Additionally, the response rate to the survey that was sent out was limited, with only 39 students responding. As the survey was meant to provide a good overview of the general student opinion, a higher response rate was hoped for. As the survey targeted the students of the TU/e, and more than 10.000 students study at the TU/e, a sample size of 100 is needed in order to get a 10% margin of error. In this study only 39 responses were registered, which means it will be very hard to conclude anything that is statistically significant out of this data. Ideally we would have reached at least 100 responses, to get a better image of the actual thoughts of the population we were sampling for.

Another important point to notice is that the few students who were interviewed were not as experienced with ChatGPT as wanted. This is however due to the limited time ChatGPT has been available to the public, and was out of our control. As ChatGPT is still very new, we can say that there are currently no expert users, only novice users. A lot of people are still trying to get a better understanding of ChatGPT and are still in the process of forming their opinion. This makes it difficult to get valuable insights from these interviews, as the users are just starting out with ChatGPT. Furthermore, it was not clearly defined what was meant by ‘experienced with ChatGPT’ which resulted in variations in experience among the students that were interviewed. Some of the students that were interviewed had used it more, and all students use it to achieve different goals. This makes it difficult to compare the interviews.

These points lead to the conclusion that the results might be inadequate to confirm or reject the hypothesis.

Future research

In the future, it would be wise to extend this study to more courses, not just the Engineering Design course. ChatGPT holds the ability to change all courses taught at the TU/e, and other other education facilities. For the TU/e specifically, the courses Calculus, Physics and Data Analysis would be a great start, as these are all courses all students of the TU/e have to follow, and courses that have the potential to be greatly affected by ChatGPT.

If this study would be conducted again for other courses, it may be necessary to use a more targeted recruitment strategy to gain more responses for the survey. Relying on snowball sampling and social circles of researchers is not enough. Using a reward, like offering money, would be a better way to get more responses.

To broaden the scope, research should be conducted into where and how ChatGPT may be used in courses, so that it is clear at the beginning of the course. Courses should be analyzed separately, to investigate for each specific course how ChatGPT can be implemented. However research into the setup of general rules for academic facilities is important to conduct as well. This should achieve transparency about the awareness of ChatGPT as well as its features and limitations.

Moreover, it might be wise to wait for more people to gain experience with ChatGPT before conducting further research. As ChatGPT continues to evolve and becomes more widely used, it is likely that more people will become familiar with it. Therefore more studies will be conducted on its potential impact on education. Waiting for more people to gain experience with ChatGPT before conducting further research will give researchers more data to work with what will lead to a more trustworthy conclusion.

In future research, researchers could consider conducting a longitudinal study to investigate the long-term impact of ChatGPT on the learning outcomes of students. The researcher would acquire the data over a long period of time. This data would include their learning outcomes which gives researchers the opportunity to analyze the long-term results of ChatGPT on educational learning goals. They could even compare the results of students who did use ChatGPT with the results of students who did not use it.

In this research, it became known that there is great need for AI detection tools. These tools are really useful for teachers, so they can check whether a student has written something by themselves or not. If it is not allowed to use ChatGPT, AI detection tools are a great way to check for this. More research has to be done into these tools, to see how reliable they are, and how they can be approved upon. Another important topic mentioned multiple times in our research exam regulations for courses need to be revised. With ChatGPT the possibility to ‘cheat’ on tests becomes more glaring, and rules and regulations need to be set into how and if ChatGPT can be used in exams. However this will also be course specific, and further research is necessary to set these rules and regulations.

Finally, as lecturers educate the experts of tomorrow, there is a need to determine how we can teach students the necessary skills for sensible use of ChatGPT. It is therefore obvious that we need to address the question of how we will integrate such AI tools into higher education, and how students will be taught in the use of ChatGPT. Sensible use needs to be taught in classes, but further research is needed to determine what the best way to do this will be.

Overall, future research should aim to use a variety of methods to obtain a better and more in-depth understanding of the potential impact of ChatGPT on education and the learning goals of courses. By doing so, researchers can provide recommendations that are based on actual results and experiences with ChatGPT.

Ethics of Chatbots and Plagiarism

An important topic to address when researching ChatGPT in education is the ethical issues related to privacy, accountability and plagiarism.

The first concern when using ChatGPT in education is privacy. ChatGPT has the ability to talk with students in a human way, which can create a false sense of security and trust. As a result, there is a risk that students may share sensitive information about their personal lives with ChatGPT. To decrease this risk, educators should take steps to establish clear guidelines and warnings around the use of ChatGPT in the classroom, emphasizing the importance of safeguarding personal information.

Another concern is that ChatGPT could provide false information to the student and the accountability of this. Because ChatGPT learns from its conversations with users, it is crucial that its responses are reviewed to make sure answers are reliable and accurate. If students are not aware of this risk, they may blindly trust ChatGPT's information, leading to them to learn and to use the wrong information for their schoolwork. In such cases, it can be difficult to determine who is accountable for the spreading of false information. As such, it is essential that educators clearly communicate the limitations of ChatGPT to students and provide guidance on how to critically evaluate its responses.

Lastly, the issue of plagiarism arises with the use of ChatGPT in education. In this research, the students who were interviewed were asked whether they thought using ChatGPT for their schoolwork should be considered as plagiarism. The general opinion came down to that if you let it complete a whole task for you, like writing a report, it should be considered plagiarism. If you use it as a way to produce a whole essay or parts of an essay, it should be properly cited, otherwise it should be considered plagiarism. Just like when someone does not report a source used for a report. However, if you’re using it as a tool to help you, it is not considered plagiarism. The teacher that was interviewed agreed with this statement. Nevertheless, it is important for educators to clearly state whether it is allowed to use ChatGPT for an exercise or course.

In conclusion, while ChatGPT has the potential to make education more efficient, it is very important that educators take a proactive approach to addressing the ethical issues that come along with its use. By being mindful of the risks related to privacy, accountability, and plagiarism, and by providing students with clear guidance and expectations, educators can ensure that the use of ChatGPT in education is both safe and effective.

Conclusion

In the presented study, we investigated the influence of ChatGPT on students in reaching the learning objectives of the course Engineering Design at the TU/e.

This report discussed the implications of using ChatGPT in Engineering Education. While these large language models like ChatGPT have the potential to revolutionize the way we access and use information, they have also raised concerns about academic integrity, student engagement, and the role of teachers in the learning process. Some of the issues that arise include the potential for cheating, the unclear definition of authorship, and the possibility of plagiarism. However, the use of AI models can also enhance student engagement, collaboration, and accessibility.

A mixed-methods approach, including a survey and interviews were used to understand the opinions and experiences of participants with ChatGPT in the context of Engineering Design. The survey was sent to university students who had completed the course, and the interviews were conducted with a professor from the Engineering Design department and frequent users of ChatGPT. The study aimed to gain insights into the frequency of ChatGPT usage, its perceived usefulness, and its impact on achieving the learning objectives of Engineering Design. The survey and interviews together aimed to provide a comprehensive understanding of the participants' opinions on the use of ChatGPT in the Engineering Design course.

The survey found that students primarily used ChatGPT for generating text in response to prompts, answering questions, and assisting with writing tasks such as essays and code. The results also suggest that students are divided on whether ChatGPT should be used in the Engineering Design course, with some seeing its potential benefits while others are hesitant. Despite these limitations and concerns, the survey results suggest that ChatGPT could be a valuable tool to complement students' learning experiences in Engineering Design. Additionally, the effectiveness of ChatGPT appears to be highly dependent on the user's skill level and familiarity with the learning objectives of the course.

Based on the interviews, student experts identified gaining information, receiving writing assistance, and saving time as the primary reasons for using ChatGPT. However, the most frequently mentioned disadvantage was its limited functionality, particularly in academic settings where it is believed to only handle straightforward tasks. In regards of the course Engineering Design, the students believe that ChatGPT could be helpful in certain aspects of the design process, such as formulating a design goal and writing a design report, defining functional and technical specifications, and executing a generic design process. However, they do not believe it would be particularly useful for more complex tasks such as selecting design concepts or developing detailed designs.

During the interview with the professor he shared his positive experience with ChatGPT and encourages his students to use it, citing benefits such as improved report writing, efficient search capabilities, and faster completion of tasks. However, he notes that ChatGPT's main limitation is its limited database and lack of creativity, which prevents it from generating innovative ideas. While ChatGPT can help with selecting design concepts and structuring concepts for detailed design, students must do the thinking and actual work themselves. Overall he found ChatGPT a useful tool but it cannot replace students' own creativity and critical thinking skills.

Comparing these results with our hypothesis, which was based on current literature and on what we thought was do-able with ChatGPT current abilities (Table 6), tells us that there is a big difference in experience in how ChatGPT can and is used. The student experts and professor had mixed opinions on the usefulness of ChatGPT, with some finding it helpful for certain tasks and learning objectives, while others found it lacking in information and unable to assist in more creative aspects. These mixed answers left us with an inconclusive result about ChatGPTs effectiveness with achieving the learning goals of Engineering Design.

Overall, it seems that the use of ChatGPT in the course Engineering Design could mainly have positive effect on achieving the learning goals, since almost all the students expressed that they would use ChatGPT as an assisting tool and not as a replacement of their own work. Its effectiveness will depend on the specific objectives and the skill level of the students using it. Further research may be needed to fully assess the impact of ChatGPT on the course learning outcomes. A certainty however is that almost no learning objectives can be completely circumvented by ChatGPT. Nevertheless, there is still a lot of debate and uncertainty about the achievability and the way students will use ChatGPT for the course.

Based on this research, the expectation is that students will try to use ChatGPT for some aspects of the course. However, usually only in a supportive role. ChatGPT is not flawless and students seem to be aware of this. Therefore the advice is to embrace ChatGPT, since it is not used for everything and it can still be a valuable addition to the curriculum of students to enhance their final deliverables. Furthermore, based on the thematic analysis, it’s clear that the professor and experts warn about the pitfalls of using ChatGPT. Therefore, the course could even go as far as to teach its students about the usability and limitations of ChatGPT, so Engineering Design will continue to teach the necessary skills to be expected from a future engineer.

Bibliography

  1. 1.0 1.1 Arimetrics (2022) What is ChatGPT. Retrieved from https://www.arimetrics.com/en/digital-glossary/chatgpt
  2. 2.0 2.1 Gao, Leo; Schulman; Hilton, Jacob (2022). "Scaling Laws for Reward Model Overoptimization". arXiv:2210.10760 [cs.LG].
  3. Chaudhry, M.A., Kazim, E. Artificial Intelligence in Education (AIEd): a high-level academic and industry note 2021. AI Ethics 2, 157–165 (2022). https://doi.org/10.1007/s43681-021-00074-z
  4. Shuai Yang & Haicheng Bai (2020). The integration design of artificial intelligence and normal students' Education. Journal of Physics: Conference Series, Volume 1453: Conf. Ser. 1453 012090. https://iopscience.iop.org/article/10.1088/1742-6596/1453/1/012090
  5. Qadir, J. (2022). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education.
  6. Limna, Pongsakorn and Jakwatanatham, Somporch and Siripipattanakul, Sutithep and Kaewpuang, Pichart and Sriboonruang, Patcharavadee, A Review of Artificial Intelligence (AI) in Education during the Digital Era (July 2022). Advance Knowledge for Executives, 1(1), No. 3, 1-9, 2022, Available at SSRN: https://ssrn.com/abstract=4160798
  7. 7.0 7.1 Lakshmanan, Lak (December 16, 2022). "Why large language models like ChatGPT are bullshit artists". becominghuman.ai. Archived from the original on December 17, 2022. Retrieved January 15, 2023. The human raters are not experts in the topic, and so they tend to choose text that looks convincing. They'd pick up on many symptoms of hallucination, but not all. Accuracy errors that creep in are difficult to catch.
  8. Beth McMurtrie (2018, August 12). How Artificial Intelligence Is Changing Teaching. The Chronicle of Higher Education(1). Retrieved from https://www.su.edu/conservatory/files/2018/09/How-Artificial-Intelligence-is-Changing-Teaching.pdf
  9. Alam, A. (2021b). Should Robots Replace Teachers? Mobilisation of AI and Learning Analytics in Education. 2021 International Conference on Advances in Computing, Communication, and Control (ICAC3). https://doi.org/10.1109/icac353642.2021.9697300
  10. Ahmad SF, Alam MM, Rahmat MK, Mubarik MS, Hyder SI. Academic and Administrative Role of Artificial Intelligence in Education. Sustainability. 2022; 14(3):1101. https://doi.org/10.3390/su14031101
  11. Kim, J., Merrill, K., Xu, K., & Sellnow, D. D. (2020). My Teacher Is a Machine: Understanding Students’ Perceptions of AI Teaching Assistants in Online Education. International Journal of Human–Computer Interaction, 36(20), 1902–1911. https://doi.org/10.1080/10447318.2020.1801227
  12. Lameras, P., & Arnab, S. (2021). Power to the Teachers: An Exploratory Review on Artificial Intelligence in Education. Information, 13(1), 14. MDPI AG. Retrieved from http://dx.doi.org/10.3390/info13010014
  13. Drori, I., Zhang, S. X., Shuttleworth, R., Tang, L., Lu, A., Elizabeth, K. E., Liu, K. X., Chen, L., Tran, S., Cheng, N., Wang, R., Singh, N. K., Patti, T. L., Lynch, J., Shporer, A., Verma, N., Wu, E., & Strang, G. (2022). A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences of the United States of America, 119(32). https://doi.org/10.1073/pnas.2123433119
  14. 14.0 14.1 van Dis, E. A., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023). ChatGPT: five priorities for research. Nature, 614(7947), 224-226.
  15. Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available at SSRN 4337484.
  16. Borenstein, J., & Howard, A. (2020). Emerging challenges in AI and the need for AI ethics education. AI and Ethics, 1(1), 61–65. https://doi.org/10.1007/s43681-020-00002-7
  17. Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1). https://doi.org/10.1186/s41239-019-0171-0
  18. Mohammad Hosseini, Lisa M. Rasmussen & David B. Resnik (2023) Using AI to write scholarly publications, Accountability in Research, DOI: 10.1080/08989621.2023.2168535
  19. Huh, S. (2023). Recent Issues in Medical Journal Publishing and Editing Policies: Adoption of Artificial Intelligence, Preprints, Open Peer Review, Model Text Recycling Policies, Best Practice in Scholarly Publishing 4th Version, and Country Names in Titles. Neurointervention. https://doi.org/10.5469/neuroint.2022.00493
  20. Kim, N. J., & Kim, M. K. (2022). Teacher’s Perceptions of Using an Artificial Intelligence-Based Educational Tool for Scientific Writing. In Frontiers in Education (p. 142). Frontiers.
  21. Williamson, B., Macgilchrist, F., & Potter, J. (2023). Re-examining AI, automation and datafication in education. Learning, Media and Technology, 48(1), 1–5. https://doi.org/10.1080/17439884.2023.2167830
  22. Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines, 30(4), 681–694. https://doi.org/10.1007/s11023-020-09548-1
  23. Lindner, A., Romeike, R., Jasute, E., & Pozdniakov, S. (2019). Teachers’ perspectives on artificial intelligence. In 12th International conference on informatics in schools,“Situation, evaluation and perspectives”, ISSEP.
  24. Popenici, S.A.D., Kerr, S. Exploring the impact of artificial intelligence on teaching and learning in higher education. RPTEL 12, 22 (2017). https://doi.org/10.1186/s41039-017-0062-8
  25. Alam, A. (2021). Possibilities and Apprehensions in the Landscape of Artificial Intelligence in Education. 2021 International Conference on Computational Intelligence and Computing Applications (ICCICA). https://doi.org/10.1109/iccica52458.2021.9697272
  26. Susnjak, T. (2022). ChatGPT: The End of Online Exam Integrity?. arXiv preprint arXiv:2212.09292.
  27. Ventayen, R. J. M. (2023). OpenAI ChatGPT Generated Results: Similarity Index of Artificial Intelligence-Based Contents. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4332664
  28. Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). Ethics in Science and Environmental Politics, 21, 17–23. https://doi.org/10.3354/esep00195
  29. Cotton, D., Cotton, P., & Shipway, J. R. (2023, January 10). Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT. https://doi.org/10.35542/osf.io/mrz8h
  30. Pimentel, C. Is ChatGPT a threat to education? For banking model of education, yes.
  31. ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? (2023). Journal of Applied Learning and Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9
  32. García-Peñalvo, F. J. (2023). La percepción de la Inteligencia Artificial en contextos educativos tras el lanzamiento de ChatGPT: disrupción o pánico. Education in the Knowledge Society, 24, e31279. https://doi.org/10.14201/eks.31279
  33. Thorp, H. H. (2023). ChatGPT is fun, but not an author. Science, 379(6630), 313. https://doi.org/10.1126/science.adg7879
  34. Zhai, X. (2022). ChatGPT User Experience: Implications for Education. Social Science Research Network. https://doi.org/10.2139/ssrn.4312418
  35. “Home | Dedoose”. https://www.dedoose.com/

Appendix

1. Survey questions

https://forms.office.com/Pages/DesignPageV2.aspx?subpage=design&FormId=R_J9zM5gD0qddXBM9g78ZOciNcKFkhVJsg7I7H6kiF5UNzNXQzhaOEpWSjZJSDg1WUZBN1lWTkkwUS4u

Research into ChatGPT

In order to review the potential of using ChatGPT in academic education, we would like to know your experiences with the technology as well as your opinion on it. We will focus on the course Engineering Design, a mandatory course where you go through a complete design process, from the first idea to the realization of the product. We will look at how ChatGPT can be used for this course.

This survey is conducted for the course Project Robots Everywhere (0LAUK0), and its results will be used to shape further investigation in academic regulations regarding AI. The results of the questionnaire will be anonymous and it will take approximately 5-10 minutes to complete.

Do you give consent to participate in this study?

  • Yes
  • No

Knowledge of ChatGPT

Have you heard of ChatGPT before?

  • Yes
  • No

Do you know the capabilities of ChatGPT?

  • Yes
  • No

Could you name some of these capabilities?

  • Open question


To give you an idea of what ChatGPT is, please read this:

ChatGPT is a large language model developed by OpenAI. It stands for “Generative Pre-training Transformer” and uses deep learning technology to generate text and respond to questions in a human way.

The model is trained on a large amount of text data, making it capable of understanding context and semantics. This makes ChatGPT a popular choice for chatbots, virtual assistants and other applications where human interaction is required.

Some examples of what ChatGPT can do:

- Write emails

- Write essays and academic reports

- Write poetry and song lyrics

- Compose music

- Answer (test) questions and solve problems

- Generate lines of code based on a prompt

- Answer customer queries

Usage of ChatGPT

Have you used ChatGPT before?

  • Yes
  • No

How often have you used ChatGPT?

  • Only once or twice
  • Monthly
  • Weekly
  • Daily

For what have you used ChatGPT?

  • Open question

How would you rate your experience with ChatGPT in general?

  • Rating from 0 to 5

How would you rate your experience with ChatGPT as an academic tool?

  • Rating from 0 to 5

ChatGPT in academic education

Have you ever used ChatGPT for academic purposes? If yes, for what course(s)?

  • Open question

Would you say usage of ChatGPT should be considered your own original work (so no plagiarism)?

  • Yes
  • No

When should the use of ChatGPT be considerd as plagiarism?

  • Open question

Looking back, would you think ChatGPT could be a good addition to the course engineering design?

  • Yes
  • No

Please, if possible, elaborate on why? (Elaboration on previous question)

  • Open question

For what do you think ChatGPT CAN be used in the course engineering design?

  • Open question

For what do you think ChatGPT SHOULD be able to be used in the course engineering design?

  • Open question

Learning objectives of Engineering Design

How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT? (Very likely - Very unlikely)

  • Execute a generic design process
  • Formulate a design goal
  • Define the functional and technical specifications
  • Generate an elaborate list of realization possibilities for the different functions of the design
  • Select a number of design concepts from an extensive list of realization possibilities
  • Make a final design choice between a number of concepts
  • Develop a detailed design that meets the specifications
  • Develop and execute a test plan for the prototype
  • Evaluate a prototype based on test results and give an advice for redesign
  • Reflect on the design and on the design process
  • Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design


Additional Learning Objectives

How likely do you think you would be able to reach the following learning objectives with the use of ChatGPT? (Very likely - Very unlikely)

  • Gaining insight in designing a simple computer program
  • Giving and understanding simple mathematical proofs
  • Understand how basic electrical and electronic circuits work


Feelings about ChatGPT

What are your feelings regarding ChatGPT? (Strongly disagree - Strongly agree)

  • I see ChatGPT as having/could have had a positive influence on my engineering design project
  • I would have seen ChatGPT as a usefull tool for my engineering design project
  • I see ChatGPT as a shortcut for tedious tasks I would have/had to perform in the course engineering design
  • I will use/would have used ChatGPT in the course engineering design regularly
  • I feel like I would have learned something when using ChatGPT for the engineering design project

2. Interview Script Student "Experts"

Interviewee:[name] Date:[date]

Interviewer:[name] Location:[location]


Introduction

Hi, my name is ___ and I’m conducting this interview as part of our project for the course Project Robots Everywhere.

The goal is to review the potential of using ChatGPT, an AI chatbot, in academic education. You have been selected because you are well known about ChatGPT and its uses, so we would like to know your experiences with this technology as well as your opinion on it.

  1. How long have you been studying at the TU/e?
  2. How long have you been using ChatGPT


Knowledge and experience

  1. For what purposes do you use ChatGPT?
  2. Why do you use ChatGPT?
  3. For which of these purposes do you find ChatGPT works the best?
  4. How do you feel about your experiences with ChatGPT?


ChatGPT in academic education

Now we will go more in depth about the potential usage of ChatGPT in academic education, and more specifically, Engineering Design.

  1. What is your opinion on using ChatGPT as an academic tool?
  2. Does using ChatGPT count as plagiarism in your opinion?
  3. Did you complete the course Engineering Design?
  4. Would you, and when yes, how would you use ChatGPT to obtain the following learning objectives of Engineering Design?
    1. Execute a generic design process
    2. Formulate a design goal
    3. Define the functional and technical specifications
    4. Generate an elaborate list of realization possibilities for the different functions of the design
    5. Select a number of design concepts from an extensive list of realization possibilities
    6. Make a final design choice between a number of concepts
    7. Develop a detailed design that meets the specifications
    8. Develop and execute a test plan for the prototype
    9. Evaluate a prototype based on test results and give an advice for redesign
    10. Reflect on the design and on the design process
    11. Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design
  1. Do you think you would actually learn these objectives while using ChatGPT?
  2. In your opinion, does ChatGPT make the education of students better, or does it hinder them in their development?


3. Interview script Teachers

Interviewee: Joris Remmers Date: [date]

Interviewer: Niels & Famke Location: [location]


Introduction

Hi, my name is ___ and I’m conducting this interview as part of our project for the course Project Robots Everywhere.

The goal is to review the potential of using ChatGPT, an AI chatbot, in academic education, so we would like to know your experiences with this technology as well as your opinion on it. We will mainly focus on the course Engineering Design and will look at how ChatGPT can be used for this course.

  1. Could you introduce yourself?
  2. What is your function / role within Engineering Design? (lecturer, tutor, etc.)


Knowledge and experience

Before we go into the specific course, we’d like to know what your knowledge and experience is regarding ChatGPT

  1. What is your knowledge and experience with the usage of ChatGPT?
    • Specifically:
      1. Design / creativity
      2. Coding
      3. Writing
      4. Summarizing / explaining (search engine)
    • How would you describe your overall experience with ChatGPT?


[Explain capabilities, show examples]

Some examples of what ChatGPT can do:

  • Write emails
  • Write essays and academic reports
  • Write poetry and song lyrics
  • Compose music
  • Answer (test) questions and solve problems
  • Generate lines of code based on a prompt
  • Answer customer queries

[Opinion on panel, experience with the course where it's used]


ChatGPT in academic education

Now we will go more in depth about the potential usage of ChatGPT in academic education, and more specifically, Engineering Design.

  1. What is your opinion on using ChatGPT as an academic tool?
  2. Does using ChatGPT count as plagiarism in your opinion?

These are the learning objectives of Engineering Design as listed on Canvas:

  • Execute a generic design process
  • Formulate a design goal
  • Define the functional and technical specifications
  • Generate an elaborate list of realization possibilities for the different functions of the design
  • Select a number of design concepts from an extensive list of realization possibilities
  • Make a final design choice between a number of concepts
  • Develop a detailed design that meets the specifications
  • Develop and execute a test plan for the prototype
  • Evaluate a prototype based on test results and give an advice for redesign
  • Reflect on the design and on the design process
  • Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design


  1. How do you expect the students to reach these learning objectives?
    1. What steps do they need to take?
    2. What tools do they need to use / are allowed to be used?
    3. How do you think chatbots like ChatGPT would influence students’ ability to obtain the learning objectives?
    4. What would you change about the learning objectives? (For example, make them more complex)
    5. Compare his answers to those of students
  1. What effect do you think ChatGPT will have on this course in general?
  2. In your opinion, does ChatGPT make the education of students better, or does it hinder them in their development?


Survey input
Figure 1. Students' answers to the question "How likely do you think you would be able to reach the following learning objectives of Engineering Design with the use of ChatGPT?"
Regarding Engineering Design (outdated):
  • 48% of students think ChatGPT could've had a positive impact on their project
  • 58% of students see ChatGPT as a useful tool
  • 54% of students would use ChatGPT for tedious tasks
  • Only 32% would use ChatGPT regularly


Theory

ChatGPT is a large language model developed by OpenAI. It stands for “Generative Pre-training Transformer” and uses deep learning technology to generate text and respond to questions in a human way.

Some examples of what ChatGPT can do:

  • Write emails
  • Write essays and academic reports
  • Write poetry and song lyrics
  • Compose music
  • Answer (test) questions and solve problems
  • Generate lines of code based on a prompt
  • Answer customer queries


Learning objectives

These are the learning objectives of Engineering Design as listed on Canvas:

  • Execute a generic design process
  • Formulate a design goal
  • Define the functional and technical specifications
  • Generate an elaborate list of realization possibilities for the different functions of the design
  • Select a number of design concepts from an extensive list of realization possibilities
  • Make a final design choice between a number of concepts
  • Develop a detailed design that meets the specifications
  • Develop and execute a test plan for the prototype
  • Evaluate a prototype based on test results and give an advice for redesign
  • Reflect on the design and on the design process
  • Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design

4. ChatGPT experts on learning objectives

Table 8. ChatGPT experts on learning objectives
Learning objective F M G N Q
Execute a generic design process Not doable Not doable No Yes No
Formulate a design goal Help with formulation Yes Yes Yes No
Define the functional and technical specifications Would be good for this Hard but possible Yes Yes No
Generate an elaborate list of realization possibilities for the different functions of the design Best done by the group Hard but possible Yes Yes No
Select a number of design concepts from an extensive list of realization possibilities Best done by the group Best done by the group Yes Yes No
Make a final design choice between a number of concepts Best done by the group Best done by the group No Yes No
Develop a detailed design that meets the specifications Help with formulation Would be good for this Yes Yes No
Develop and execute a test plan for the prototype Probably not doable Can help but not detailed Yes Yes No
Evaluate a prototype based on test results and give an advice for redesign Help with formulation More effort than normal No Yes No
Reflect on the design and on the design process Would definitely use Not doable Yes No No
Write a design report describing the design process, the foundation of the choices made and the evaluation of the delivered design Would definitely use Only for parts of the report No Yes No

5. Link to panel discussion:

https://tue.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=55f2e76b-58d4-4397-a78a-afce00c47793

6. Approach, milestones and deliverables

Planning

Here follows a Gantt chart of all our deadlines to be finished at 12pm on Sunday of the corresponding week. The letters indicate the group member responsible for making the deadline and the scheduling for the specific task.

Table 9. Gantt chart
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8
Approach, milestones and deliverables Ni
Problem statement and objectives Q
Users and requirements G
Summaries of literature F
Literature review M
Surveys Na
Surveys analysis All
Interview preparation teachers All
Interview preparation students Q
Organize interviews teachers Ni
Organize interveiws students All
Conduct interviews teachers Ni/M/F
Conduct interviews students All
Interview analysis All All
ChatGPT panel All
State of the art F
Advise statement Na Na
Discussion F
Future research Q
Reflection ? Ni
Finalization of wiki M
Presentation Q
Chair meetings N
Keep wiki up to date M
Minute taker Q F G Na M Q F

Approach

  • Literature study
  • Interviews, surveys

Deliverables

  • Advice for future