Is New York City’s AI chatbot providing risky advice to local businesses?

New York City’s AI-powered chatbot, designed to assist business owners with navigating government regulations, has come under fire for providing false and illegal advice, as revealed by an investigation conducted by The Markup.

Launched in October 2022 as part of Mayor Eric Adams’ “New York City Artificial Intelligence Plan,” the chatbot, powered by Microsoft’s Azure AI services, was intended to support small businesses in understanding and complying with various regulations.

However, The Markup’s report uncovered numerous instances where the chatbot provided inaccurate information on crucial topics such as housing policies, worker rights, and rules for entrepreneurs. For example, the chatbot incorrectly stated that landlords were not required to accept tenants on rental assistance, despite the city’s own website confirming the illegality of such discrimination.

Furthermore, the chatbot erroneously advised that bosses could take a cut of their worker’s tips, in violation of New York’s labor laws prohibiting such actions.

Rosalind Black, Citywide Housing Director at Legal Services NYC, tested the chatbot herself and found more inaccuracies related to housing laws, leading her to label the chatbot as “dangerously inaccurate.”

Andrew Rigie, Executive Director of the NYC Hospitality Alliance, echoed these concerns and emphasized the urgent need for the chatbot to be rectified to prevent further misinformation.

In response to the investigation, Leslie Brown, a spokesperson for the NYC Office of Technology and Innovation, acknowledged the chatbot’s shortcomings and assured that efforts are underway to enhance its accuracy and reliability to better support small businesses in the future.