Better understanding the risks of AI for New York City agencies

As chair of the New York City Council Technology Committee, Brooklyn Council Member Jennifer Gutiérrez has grappled with how to oversee city agencies that are currently utilizing artificial intelligence as well as those barring the technology. Gutiérrez has a particular interest in technological advancements in education and held a joint hearing on emerging technology and data security in the city’s public school system in September and is planning future hearings with the Office of Innovation and Technology. This interview has been edited for length and clarity.

What were the big takeaways from your recent joint hearing on AI in schools? 

At the hearing we only had representatives from New York City public schools. We didn’t have (the New York City Office of Innovation and Technology) present. I wished we would have had them present answering questions.

We learned a couple of things. One is I feel that the momentum is reactionary right now around AI technology. Having the chancellor and the whole New York City public school system go back on their ChatGPT ban is an example of the way the city is reacting to this technology. Schools still have to get permission to utilize it in the schools, but it’s there already. The reality of how quickly technology is integrated into our everyday life and the reality is our New York City public school system needs to set up much better protocols to integrate this in our schools. I think they are trying, but I don’t think they’re moving fast enough.

We focused on safety with respect to data breaches that they experienced. We always want to center that but there are instances where technology and security can exist.

What were some of the concerns that the Department of Education had that you heard at the hearing?

They are just now putting together a roundtable to put together a policy on AI and that could be a few months. They emphasize that they agree that this could be a tool. What they felt could be banned initially they understood could be useful. The feedback they heard from educators is they are not sufficiently trained and supported for these tools. They need to vet these tools and make sure they are useful. A lot of these technologies are at the individual discretion of these schools, and I did not feel confident there’s a place where all these tools live.

And the last part, to ensure that families and staff are able to use these tools at home. For example, when ChatGPT was banned, they did not prevent students from using it at home. The scenario is that they would be using ChatGPT at home with homework and parents would not be able to implement safeguards for their devices or understand how their children would use technology to help with their homework. That was the feedback that parents were not being informed that children were using it at home, or not feeling equipped to help them if they needed to. If a parent were not up to speed, then that could be a challenge.

In order for this technology to be a useful technology for our system overall, we had to bring family, parents and caregivers along, especially in communities that didn’t have devices. During the pandemic, the Department of Education did a good job helping students bring devices home with them and giving them hot spots and connectivity. There are instances where families don’t have the digital literacy to help their children use the device.

They also acknowledge that they are lagging on having public school graduates equipped with the right training and access with technology to start filling into these jobs. Of the tech jobs, only 8% are filled with New York City school graduates. That’s a really important piece they acknowledge. They need to be doing more, it’s really just lagging.

Do you know of any other progress the city has made in its efforts to develop and implement AI strategy beyond the schools?

I don’t. This would have been a really good instance where the Office of Innovation and Technology could have spoken to this citywide since they contract and work with every single agency. I don’t have a good sense of how these agencies are utilizing this. Someone asked if the council is working on it, but no, I don’t think we are. It’s truly the reality of how this industry is moving. We had Manhattan Borough President Mark Levine come and testify and AI can be very useful for identifying trends, help train folks and help bridge the gap to language access issues.

What are the ways that the city government is utilizing AI currently, and what are the ways that the city government could utilize AI in the coming years?

There are a couple examples where they already are using AI that are not in the generative ChatGPT space. It’s something we’ll begin asking OTI at our hearings. There are instances where they can use predictive analytics to help certain agencies do this work. I’m sure OTI uses it a little but, we want to have a better understanding on how they use generative AI. It’s something they haven’t been clear about, but there are opportunities to use it. 

On the ground we know they’re using it. I heard from colleagues around the government, “Hey we’re messing around with it, we’ll see how far it gets.” But we don’t have a formal conversation with agencies to understand how each agency is using AI.

What city legislation should be passed or regulations implemented around the use of AI in government or in general?

We have a lot of lessons to learn from the writers strike and what they’re asking for regarding AI. That’s an opportunity. We want to work with schools that have used AI successfully to see some of the challenges and lessons learned. John Jay College used AI to identify students that needed help to increase their graduation rate. And we have some really good advocates. Borough President Levine is enthused and working on this stuff. We’re looking through that now to see how we can legislate some of this.

The core of it is we want this to operate in a safe way. We don’t want to compromise safety, but there are opportunities. We want to make sure we’re being smart, innovative and equitable. We want to expose New Yorkers to this and make sure we’re graduating more Black and brown New Yorkers into this field. We’re breaking all of this down to see how we can create a system, protocol and legislate it.

This hearing got some attention. We see what other cities are doing on how AI can improve traffic safety and constituent services in government offices. There are a lot of opportunities. We will be asking agencies to do that. At that hearing we heard one of the bills (Education Committee) Chair (Rita) Joseph is sponsoring, a resolution making sure that (the Department of Education) mandates training on generative AI. That’s one bill we heard that I hope will pass.