Comet Report Reveals Unique Challenges, New Reality for ML Practitioners 

Comet, the maker of a machine learning development platform, has released a new survey report that found ML practitioners currently have a lot to contend with.

“State of MLOps Industry Report: 2023 Machine Learning Practitioner Survey” is the company’s second annual survey on the state of ML, conducted by Censuswide, that explores the issues affecting the adoption of ML as well as project and initiative success. Key topics emerging from this year’s report include the AI Bill of Rights, bias, and operational challenges amid tightening budgets.

Comet CEO and Co-founder Gideon Mendels says that ML practitioners are facing a new reality with a unique set of challenges ahead. Even though organizations may be spending less due to the current economic uncertainty, Mendels says that because of ML’s potential to unlock business value, there may be a push for ML practitioners to tackle more complex problems quickly. Addressing bias or adhering to the AI Bill of Rights means organizations must have the proper tools in place in order to ensure the success of their ML projects, Mendels says.

The U.S. White House Office of Science and Technology Policy recently published its “Blueprint for an AI Bill of Rights,” a framework that names five principles that should guide the design, use, and deployment of automated systems. Though these principles aim to protect the American public from unsafe or ineffective systems, AI bias, and data privacy issues, their effects on AI and ML projects will be impactful.

Comet’s report gauged the reaction of the ML community, finding that 73% of practitioners agree that the AI Bill of Rights (BOR) should be mandatory by law versus opt-in. Mendels believes the ML professionals who disagree with making the bill mandatory may be concerned about the additional steps and governance involved that could further slow down the already lengthy process of getting models into production.

“Although there appears to be support for the BOR, approximately 39% of individuals still believe that it will impede their approach to ML deployment and development by slowing down the process. In terms of negative impacts, nearly 37% believe that the BOR will complicate the process of ML deployment and development, while 35% believe it will make the process more costly,” Mendels told Datanami in an email interview.

(Source: Comet)

“However, there are also positive impacts, as roughly 38% of individuals believe that the BOR will increase the safety of the ML deployment and development process, and a similar percentage (37%) believe it will decrease the likelihood of privacy violations,” he said. “Lastly, over 35% of individuals believe that the BOR will reduce the occurrence of unsafe or ineffective ML systems.”

As far as bias in AI products is concerned, the report notes that some view bias as overhyped and that ML practitioners are capable of implementing best practices for mitigation. Others believe it will be a continued problem for AI systems, and the report shows that 35% of those surveyed think the BOR will reduce the frequency of unsafe or ineffective ML systems. In fact, 38% have a designated point of contact or support team that is looking out for bias when planning the design and/or launch of an AI-enabled product. An additional 33% of respondents said that reducing the risk of bias is a main benefit of explainable AI.

Over a quarter (27%) of those surveyed believe that bias will never truly be removed from AI-enabled products, and a further 8% are unsure. “Present-day machine learning models heavily rely on data for their training. In most scenarios, this data is not freshly generated and instead comes from existing databases,” Mendels says. “However, there may be existing bias in this data when it comes to how loans have been denied for certain sections of society. Using this data to train a model will only propagate the existing bias.”

Mendels says that although data scientists can manually remove this biased data, the bias could re-enter the model after real-world deployment or it could learn patterns from other vendors. “Therefore, it is imperative to continuously validate new data and remove bias from the model. However, despite these efforts, it may not be possible to eliminate all bias, and a small amount of it may still exist,” he said.

Another key area the report explores is the additional ML challenges brought on by the state of the economy. Mendels notes that 100% of the ML practitioners surveyed said the economic situation will impact their business in some way, and the most common impacts that respondents expect are redundancies in the tech team (40%), followed by an impact on budgets (37%) and a hiring freeze (36%).

“In the event of an economic downturn, companies will cut on hiring new ML practitioners. This could make it more challenging for projects to get the right talent to work on various stages of the ML lifecycle,” Mendels said. “As a result, data scientists might be forced to work on Data Preparation or MLOps which is not their strength, and they might be focusing less on building good models. This will exacerbate existing challenges around deploying good ML models to production.”

Changes in funding could also affect research efforts for ML projects in academic and industrial sectors, along with available venture capital for startups, Mendels says, which could cause a slowdown in innovation. Mendels predicts there will be more pressure on data scientists and ML practitioners to show ROI through deploying more models into production and showing quick business value with already deployed models.

The looming possibility of these economic stressors led 32% of respondents to say that innovation will slow as a result. The survey also uncovered other anticipated challenges, including sustainability (41%), followed by retention (39%), hiring staff with correct institutional knowledge (36%), and explainable AI (36%).

Comet’s report revealed many other insights. To read the full “State of MLOps Industry Report: 2023 Machine Learning Practitioner Survey,” visit this link.

Comet will be hosting an upcoming conference, Convergence 2023, on May 7-8. The virtual event will feature over 25 talks, panels, and workshops by leading data scientists.

“We have a great lineup of diverse speakers across hardware and software, and I am also looking forward to hearing female leaders’ points of view,” Mendels said. “We will be covering a variety of topics that appeal to both practitioners and to business leaders, [with topics] like ‘Designing and Operationalizing Responsible ML,’ ‘Role of Data Scientists in the Age of GPTs and LLMs,’ and ‘Data Storytelling.’”

This article first appeared on sister publication Datanami.