The EU AI Act: Is Your AI Under Control?

"AI under Control" refers to the principle that AI systems should not operate without human supervision. Humans should always have the ability to understand, direct, and intervene in the actions of AI systems. This principle ensures accountability, transparency, and safety in AI systems. Without it, we risk creating AI systems that make decisions we don't understand, can't predict, and can't control. The importance of this principle is underscored by emerging legislation, which sets out stringent requirements for AI systems. 

Being in control of your AI is not a one-step process but a journey: 

Testing and Validation: AI systems should undergo rigorous testing and validation processes to ensure they function as intended and do not produce unexpected results. This involves creating diverse test scenarios and using robust validation techniques to ensure the AI system's decisions are accurate and reliable. This is crucial in maintaining control over AI systems and ensuring their reliability. 

Human Oversight: There should always be a human in the loop who can understand and intervene in the decisions made by an AI system. This human oversight should extend throughout the lifecycle of the AI system, from development to deployment and maintenance. It involves training personnel to understand the AI system's workings and providing them with the tools and authority to intervene when necessary.  

Clear Accountability: Who is responsible for the decisions made by an AI system? The answer should be crystal clear. This includes both the individuals who develop and deploy the AI system and the organizations that use it. Precise accountability mechanisms, such as detailed documentation and traceability of decisions, should be in place.  

Regular Reviews: AI systems should be regularly reviewed and updated to ensure they continue to operate as intended. This includes monitoring the system's performance, updating its algorithms, and retraining it with new data as needed. Regular audits should be conducted to ensure the AI system continues to meet its intended objectives and complies with all relevant regulations.  

Transparency and explainability are cornerstones of "AI under Control". However, achieving them is not straightforward. Many AI systems, particularly those based on deep learning, are often described as "black boxes" because their internal workings are not easily understandable by humans. 

Explainable AI, including surrogate models, has been proposed to solve this problem. These models aim to simplify the complex workings of AI systems and make them understandable to humans. However, there's valid skepticism about their effectiveness. While surrogate models can provide a simplified explanation of an AI system's decisions, they may not accurately represent the complex computations of the original model. This could lead to oversimplified or even misleading explanations. Therefore, developing more effective methods for explaining AI decisions and ensuring transparency is crucial.  

Safety is a key concern in AI systems. These systems should be designed and tested to ensure they do not pose risks to people or systems. However, predicting all possible scenarios an AI system might encounter is challenging, making it difficult to ensure complete safety. This involves conducting thorough risk assessments, implementing safety measures, and continuously monitoring the AI system's operation to detect and mitigate any potential risks. It also involves preparing contingency plans for when things go wrong, ensuring safeguards are in place to prevent harm.  

AI systems should be designed and used to respect ethical principles, such as fairness and non-discrimination. However, programming complex ethical rules into an AI system is challenging, and there's a risk that AI systems could inadvertently make biased or unfair decisions. This requires a careful design of AI systems, taking into account ethical considerations from the outset and implementing mechanisms to detect and correct any potential biases in the AI system's decisions. It also involves fostering a culture of ethical AI use within organizations, ensuring that everyone involved in developing and deploying AI systems understands and adheres to these ethical principles.