New York City schools are embracing AI. Lawmakers had some questions.

Are New York City schools ready for the AI revolution?

At a joint City Council hearing hosted Wednesday by the Committee on Technology and the Committee on Education, city Department of Education officials testified that artificial intelligence tools, and even generative AI tools – a more advanced technology that includes ChatGPT – were important to students’ education and career training. “Our mission is to ensure that each student graduates on a pathway to a rewarding career, long-term economic security, equipped to be a positive force for change,” Melanie Mac, director of the Office of Student Pathways, said during opening remarks. “To pursue this mission, New York City public schools is aspiring to be a global leader in embracing AI, and expanding existing programs that build computer science and digital fluency skills as essential concepts layered across our core curricula and subject areas.”

New York City Council members had some questions about what exactly that meant. “Because of AI’s tremendous potential to benefit those in education, it is essential to ensure its ethical use, and that it enhances rather than detracts from the educational experiences of our city’s students and teachers,” said Council Member Jennifer Gutiérrez, chair of the Technology Committee.

Lawmakers also sought clarity on what kinds of AI tools were already in use. While the Department of Education briefly banned ChatGPT in the previous school year, it reversed that decision and now has the tool on a list of restricted sites that individual schools can request to unlock. Since that about-face, the department has partnered with Microsoft to pilot an AI-powered teaching assistant that can answer students’ questions. Even before generative AI, other AI and machine learning tools like speech recognition and language learning tools were being used, Mac said.

In the wake of a pair of large-scale data breaches that compromised current and former students’ data, lawmakers also had questions about how AI products – and third-party technology vendors in general – are vetted by the department and individual schools. Some council members who are public school parents called for more transparency in what tools are used. “I’ve received countless letters about breaches of my child’s data, and the responsibility for me now to respond and monitor my child’s data, based on a company that I never made the agreement with to begin with,” Council Member Alexa Avilés said. Republican City Council Member Vickie Paladino – Avilés’ ideological opposite in many ways – echoed her concern.

Department officials said any software – whether it’s a purchased through a centralized contract or acquired by individual schools – had to go through the same vetting process, which includes cybersecurity reviews and collaboration with the Office of Technology and Innovation. A report from the state comptroller in February found that the department didn’t mandate steps to determine if AI tools available to schools had been evaluated for potential biases.

Council members may get more clarity on their questions over the next year, as the department develops a K-12 AI policy toolkit, including considerations of responsible and ethical use, which is intended to be finalized by June 2024.