NYC’s AI chatbot caught advising businesses to break the law, city refuses to remove it

The artificial intelligence-powered chatbot created by New York City to assist small business owners is facing backlash for providing bizarre and inaccurate advice, including misstating local policies and encouraging businesses to break the law. Despite these issues being brought to light by tech news outlet The Markup, the city has decided to keep the chatbot on its official government website.

Mayor Eric Adams defended the decision, acknowledging that the chatbot’s responses were incorrect in some areas. Launched in October as a resource for business owners to navigate the city’s bureaucracy, the chatbot includes a disclaimer warning that it may produce incorrect or biased information and that its responses are not legal advice.

Critics, including computer science professor Julia Stoyanovich, have raised concerns about the lack of oversight and accountability in deploying AI-powered systems like this chatbot. The bot has been found to provide false guidance on various topics, from employment law to waste management, and even gave absurd advice such as serving cheese nibbled on by a rodent in a restaurant.

Microsoft, which powers the chatbot through its Azure AI services, stated that they are working with city officials to improve the accuracy of the responses. Despite the criticism, Mayor Adams defended the technology, stating that addressing issues is part of refining new technology.

Experts warn that the pitfalls of New York’s chatbot should serve as a cautionary tale for other cities considering similar AI systems. The need for accountability and accuracy in providing information to the public is crucial, especially when it comes to government-backed initiatives like chatbots.