Photo By: Towfiqu barbhuiya
AI is becoming increasingly involved in decisions that were once made entirely by people. Businesses are using AI to analyze information, identify potential security threats, answer questions, recommend actions and automate routine tasks.
As these systems become more capable, some are also being given the ability to act with less direct human involvement.
That raises an important question: How much should organizations expect AI to understand before allowing it to make decisions on their behalf?
“We need to remember that since AI does not have the same context nor experience that a human has, we cannot expect it to act in the way a person would,” said Melissa Cohoe, Global Strategist for Security, Risk & Resilience at NewRocket.
The distinction between what AI can process and what a person can understand may become increasingly important as companies rely on the technology for more sensitive tasks.
AI can process information, but context matters
AI systems can work through enormous amounts of information in a short period of time. They can identify patterns, compare data points and flag activity that appears unusual.
That can be extremely useful, particularly in areas such as cybersecurity, where organizations may have thousands of alerts and events to review.
But identifying something unusual is not the same as understanding why it happened.
Imagine an employee suddenly logs into a company system from another country. An AI system might identify the login as suspicious because it differs from the employee’s normal behavior.
A person investigating the alert might know that the employee is traveling for work. They might also know that the employee is working with a particular customer or is responding to an incident.
That context could completely change the decision.
The AI did not necessarily make a bad assessment. Based on the information available to it, the activity may genuinely have looked suspicious. The problem is that the system may not have had access to everything a human knew about the situation.
The risk of automated decisions
The difference becomes more significant when AI moves from making recommendations to taking action.
An AI system could be designed to respond automatically to a suspected security threat. It might disable an account, block network access or isolate a device.
Automation can help organizations respond quickly. In cybersecurity, speed can be critical when an attack is underway.
But an incorrect action can also create consequences.
If an employee’s account is disabled during a critical business operation, for example, the response could interrupt work or prevent someone from dealing with an actual security incident.
A human might recognize that the situation requires more investigation before taking action. An AI system may simply follow the rules it was given.
This is one reason organizations need to think carefully about which decisions should be automated and which should involve human review.
Experience is difficult to replicate
Human judgment is not based only on the information directly in front of someone.
People build knowledge over time. They remember previous incidents, understand organizational relationships and learn how systems and processes behave under different circumstances.
That experience can influence decisions in ways that are difficult to capture in data.
An experienced security professional may know that a particular alert is usually harmless. They may also recognize a combination of small details that suggests a much more serious problem.
AI can be trained and given access to large amounts of information, but that does not mean it has the same understanding of a situation that a person develops through years of experience.
This is especially relevant as businesses experiment with AI agents that can perform multiple tasks and interact with other systems.
The more independently an AI system operates, the more important it becomes to understand what information it has, what information it lacks and how it will respond when a situation does not fit neatly into its instructions.
Humans still have a role
The answer is not necessarily to keep humans involved in every AI decision.
If a person had to approve every action taken by an AI system, much of the value of automation would be lost. There are many routine decisions that AI can likely handle efficiently.
Instead, organizations may need to determine where human judgment provides the most value.
Low-risk, repetitive tasks may be good candidates for automation. More complicated decisions involving security, privacy, financial consequences or critical systems may require additional oversight.
That could mean having a person review certain decisions, establishing limits on what an AI system can do or requiring the system to ask for human approval when it encounters an unfamiliar situation.
The goal is not necessarily to prevent AI from acting. It is to make sure the technology’s ability to act is matched by appropriate safeguards.
Knowing what AI does not know
Much of the conversation around artificial intelligence focuses on its growing capabilities. AI can write, analyze, summarize, predict and automate tasks that once required significant amounts of human time.
But capability has limits.
An AI system does not automatically understand an organization’s history, an employee’s intentions or the circumstances surrounding an unusual event. It can only work with the information and instructions available to it.
That makes Cohoe’s point particularly relevant as organizations give AI more responsibility.
The challenge may not be simply deciding whether AI is capable of performing a task. Companies also need to consider whether the system has enough context to perform that task safely.
AI can process information at remarkable speed. Humans bring experience, judgment and an understanding of circumstances that may not appear in the data.
As AI becomes more involved in business and security decisions, the most important question may not be how much work people can hand over to machines.
It may be knowing which decisions should never be handed over completely.