It’s 10 am.

The server farm has been working at 100% all night, processing the latest changes into the AI core programming that were suggested by the AI core itself.

You sip your… 8th? 10th? cup of coffee. Who is counting now.

If this experiment works, it will be the last time we, humans, need to to perform upgrades in the AI core directly.

Not that we are doing the programming anymore. For a few years now, the algorithms and data processing have become too complicated for any human to handle by himself.

We “prompt” Narrow AI systems to prepare the updates, according to our new design ideas and creativity, but employing the raw power of the AI models we have so painstakingly trained.

But this time… this time is different. The AI core proposed the new design by itself.

If it proves successful, and we plug the AI core into itself, it will be able to perform its own upgrades.

No humans required. No sleep. No coffee.

Who can really predict how many… weeks? days? hours? may it take to upgrade itself again. And again. And again.

One wonders what will mean to become the 2nd most intelligent species in this planet.

We’ll find out.

In today’s issue:

• The Singularity is Near: What is the Singularity and why should we care.

• Achieving and Surpassing the Computational Capacity of the Human Brain: The road ahead.

• The Impacts and Existential Risks of the Singularity: Should we be worried or excited?

• And more…

This week’s useful finds:

• Inside Large Language Models: How AI Really Understands Language: An excellent explanation of how your favorite Generative AI tool (ChatGPT, Grok, etc.) works.

• The publishing industry is broken, but it doesn’t have to break us: Really insightful discussion on strategies for business growth. It is mainly focused on the publishing industry, but applies for every Small Business model I can think of.

• 10 ways Blockchain can help Small Businesses: An interesting article on the use of Blockchain technologies for Small Business growth, written by the world leader on Logistics, DHL.

The Singularity

You might have already heard the term Technological Singularity. Or just Singularity.

Or perhaps you haven’t. So we will revisit it here.

What is it? Where it comes from? Is it possible? Are we far from it? Near?

According to Wikipedia1, the Technological Singularity is the moment when our human technology advances so fast, and almost by itself, that it becomes impossible to stop, with unforeseeable consequences.

The first person to use the term was John von Neumann in 1958, one of the fathers of the Digital Computers, and inventor of the von Neumann architecture that is currently used by most processors in the world.

But perhaps the responsible of the wide circulation of the term is Ray Kurzweil, author of “The Singularity is Near2”.

“What, then, is the Singularity? It's a future period during which the pace of technological change will be so rapid, its impact so deep, that human life will be irreversibly transformed. Although neither utopian nor dystopian, this epoch will transform the concepts that we rely on to give meaning to our lives, from our business models to the cycle of human life, including death itself. Understanding the Singularity will alter our perspective on the significance of our past and the ramifications for our future. To truly understand it inherently changes one's view of life in general and one's own particular life.”

— Ray Kurzweil

Linear vs Exponential Development

If I tell you that by 2045 we might have already reached the Singularity, with an Artificial Intelligence more capable than the Human Intelligence, you’ll probably say it is too soon.

But, for all we know, it is very probable we will achieve it by then.

The reason why it is hard for us to grasp how close we are is because of the linearity of the human mind thinking.

We perceive time linearly, and we tend to use the past as a predictor for future events.

And it works pretty well when the events happen in a linear progression.

But technology advances exponentially.

So, if we use the same linear mindset to try to predict its future, we will fall short.

Greatly.

As an example, look at the previous figure, where we pit together a linear and an exponential curve.

Notice how at the beginning they move almost together, but at some point the exponential curve starts to move away at an ever accelerating pace.

Just think what we have achieved during the last 10 years in telecommunications, health, electronics, materials, etc.

And how much time it took to get a similar level of advancement during the 20 previous years.

The pace is accelerating, and soon we will see advancements in 5 years that previously might have taken centuries.

That’s the power of the Exponential Development, and we are right inside of it.

The Six Epochs

The pace of development is getting faster every day, but to get where we are now we had to go through a lengthy and slow process.

Kurzweil proposes six epochs for understanding where we come from and where we are going:

Epoch One: Physics and Chemistry

From the Big Bang and the beginning of the Universe, we have the lengthiest Epoch, where matter, atoms and molecules were formed into what we have now.

Epoch Two: Biology and DNA

Then, at some point, life appears.

Probably one of the most amazing developments was the appearance of DNA, which allows to store enormous amounts of information, enough to describe and build life itself.

Epoch Three: Brains

Soon, living beings begun evolving central nervous systems and brains.

This gave place to the appearance of Intelligence, and soon, Human Intelligence.

Epoch Four: Technology

We started creating a culture around us, and with it, Human Technology.

From fire, the wheel, and agriculture, all the way to our latest developments in Artificial Intelligence and Nanotechnology.

This period has been the shortest so far, but the next ones promise to be even shorter.

Epoch Five: The Merger of Human Technology with Human Intelligence

We are already on the rollercoaster towards the unification of our technology with our intelligence.

Every day we depend more and more from the tools we build: food from farms, vehicles, energy production, computers, telecommunications and more.

We already know the concept of “wearable technology”.

Our smartphones makes us one with the Internet, even if it happens rather inefficiently (by typing, talking, reading and watching).

Soon we will start interfacing directly with our technology, so the relationship becomes more efficient.

We can already see the results of these efforts, through companies like Elon Musk’s Neuralink3.

But our truly defining moment will be when we finally achieve the old dream of building the Artificial General Intelligence.

Right now we can already consider ourselves a Human/Technology civilization.

Under no circumstances we could have achieved what we have now without the use of previously developed tools and technology.

And we continue to do so.

But once we have Artificial Intelligence comparable (or superior) to the Human Intelligence, we will start collaborating with our technology, not only using it.

And the Artificial Intelligence will start developing its own tools, and developing itself, well past the human biological limitations.

The Singularity.

Epoch Six: The Universe Wakes Up

This will mark the true expansion of Humanity into the cosmos.

Humanity and its technology will become one, and our resources and energy requirements will skyrocket.

Following Kardashev’s scale4, we will need first to control all of the energy that reaches the Earth from the Sun, then become able to utilize all the energy produced by the Sun itself, and finally all the energy produced by our galaxy.

Achieving and Surpassing the Computational Capacity of the Human Brain

But to reach the Singularity, we still need to tackle several problems and develop some technologies and new paradigms.

One of them is dealing with the computational capacity to support an Artificial General Intelligence.

Human brains and current computers work in very different ways, and have very different advantages and disadvantages.

Our brains are a huge neural network capable of incredible pattern matching and recognition, but slow for some computational processes and of limited memory.

Current computers are extremely fast, but work sequentially and we are still trying to build proper neural networks with them.

Nevertheless, computers can communicate with each other at much faster speeds than our brains, retain information indefinitely, and once one learns how to do something, it can be transferred immediately to others.

So we need to develop computing techniques capable of harnessing the best of both worlds, while avoiding the pitfalls.

Reverse Engineering the Brain

Enormous amounts of resources have been devoted to build the current Artificial Intelligence attempts.

Those can be classified as Narrow Artificial Intelligence, because they are focused on specific tasks and incapable of general reasoning (check my previous article5 where I explain this in the context of Customer Experience AI).

But the real effort to get an Artificial General Intelligence (AGI) is still happening.

We need yet to understand how the brain works, not only at the biological level (think of it as hardware) but also at the reasoning and processing level (what would be the software).

This will shed some light on what are we trying to replicate, even if we don’t build it exactly in the same way it works in Nature.

In the meantime, all those specialized Artificial Intelligences built on top of Nvidia processors, soaking up enormous amounts of energy, will become tools that will help us achieve the Real Artificial Intelligence.

The Impacts

The exponential development of our Technology is caused by the interaction of many science disciplines helping each other in a virtuous cycle (or spiral).

Our newest attempts in Narrow Artificial Intelligence (ChatGPT, Grok, Claude, etc.) are already helping scientists to innovate in their own fields.

Better and faster biology research (proteins and amino acids) result in better treatments and health, making us live longer.

Narrow AI is helping to create better materials (nanotechnology) and design better processors.

Those enhanced processors will help AI to run better, and help us build better robotics and hardware to help us everyday.

So, we have Artificial Intelligence, Nanotechnology, Robotics, Genetics, etc. helping each other in an ever increasing virtuous spiral that will lead us into the Singularity.

We will experience in the next few decades the impacts of this vertiginous rate of development.

It will change the Human Body and the Human Brain, to make them more efficient and healthy, even before the Singularity itself.

Our longevity will be greatly increased by those developments, making us more productive as a species.

Our productivity will also benefit from better ways of learning (remember Neo, from The Matrix, and his famous “I know kung fu”), and our way of working.

But we will also be impacted by new risks, like the use of Artificial Intelligence for warfare (like Lethal Autonomous Weapon Systems).

Existential Risks

When we talk about “Existential Risks”, we are not just talking about some events or their consequences that can cause a finite or manageable amount of damage.

We are talking about World Ending events. All Life Ending events.

And with Artificial Intelligence, Genetics and Nanotechnology, those leave the realm of dystopia science fiction, and become a very possible outcome.

As technology and science advance, those risks appear around us even more frequently than we care to imagine.

For example, a few years ago, the CERN Large Hadron Collider (LHC) performed an experiment that some people believed could create a micro Black Hole.

Of course, this was an exaggeration, because even if the black hole were created, it would be so minuscule that no danger could come from it.6

But it shows how we are starting to dabble into matters that could easily mean destruction at the cosmological level.

Think of the first nuclear bomb. Before that, it would have been unthinkable that we, as a species, would be able to end all life on Earth.

Now we are very capable of such destruction.

Genetic manipulation of viruses, fungi and bacteria can create strains so lethal that would wipe most of the human population.

But it is a well known fact that a disease can hardly wipe an entire species, because of the high level of variability in our immune systems and genetics.

So, at least part of the human species would survive.

Now think about a nanotechnological product gone wrong.

One that was designed to disassemble certain types of waste or materials, and with the ability to replicate itself as needed to cope with larger amounts of required work.

An original programming defect, or a defect introduced due to faulty replication, makes it start disassembling everything around: living and inert.

This is what Ray Kurzweil aptly named “The Grey Goo Scenario”.

Everything on Earth would become that grey goo byproduct of the faulty nanotech process gone rogue.

It could wipe all life in as little as two weeks, and we would be completely unable to do anything.

Our immune systems have not evolved to deal with something like this.

And let’s not get started with the “Terminator” scenario, where a superior Artificial Intelligence, coupled with advanced Robotics decide to get rid of the human species.

So, what can we do as a species to annihilate ourselves while trying to reach higher levels of technology?

Defenses

We can plan and build defenses for dealing with those Existential Risks before deploying the technologies that could trigger the undesired events.

Like bolstering our immune systems through more powerful vaccines and gene therapies, capable of dealing with unknown biological threats.

When dealing with the Grey Goo, we could create an artificial immune system, integrated not only to our human bodies, but to the whole environment.

Only defensive nanotechnology would be able to stop a rogue or faulty nanotechnology from turning everything into ashes in a matter of days.

And we are already trying to infuse Ethics and Human Morality into our early attempts at Artificial Intelligence, with varied results.

As our understanding of the Human Thought Process, and Human Intelligence increases, we will be able to try to replicate it using technology.

And, hopefully, we will be able to teach it “how to be good” too.

Relinquishment

Building defenses is not the only way to deal with the Existential Risks.

Just a year ago, during the height of the Generative Artificial Intelligence hype, a group of renowned Computer Scientists, Artificial Intelligence Experts were terribly worried on the perceived recklessness of our AI Gold Rush.

They signed an open letter named: “Pause Giant AI Experiments: An Open Letter”.7

The idea was to convince the major AI players to stop their developments until some guard rails could be agreed and established.

The fear of inadvertently giving birth to the first Artificial General Intelligence, that would be capable of enhancing itself and surpassing humans in weeks or perhaps days, was the main driver.

An all powerful Artificial General Intelligence without any regards of human values, ethics and moral could very well mean the end of the human species too.

So, Relinquishment is another option for dealing with technologies that could pose Existential Risks.

Temporary or permanent relinquishment.

Unfortunately, this option entails the danger of rogue agents (private or governments) ignoring the relinquishment accords, and developing the forbidden tech underground.

And ultimately this would be even more dangerous, because without “the good guys” trying to develop defenses, any mishap would end in disaster.

Regulations

And, finally, we could try to create strong regulations to disincentivize the technology players from following paths that can become harmful in the long run.

The European Union is following this idea, as you can read in my article “Prohibited Artificial Intelligence Practices in the European Union”.8

But this approach also has important drawbacks.

While the European Union establishes stiff regulations to the research and development of AI, other economic blocs, like China and the US don’t.

This entices researchers to migrate to those territories where their work isn’t entangled by red tape, effectively voiding the intent of the regulations in the first place.

So, unless there is a general agreement on the regulations applied to technologies that pose Existential Risks, the enforcement of those won’t solve the issue.

Conclusion

I hope this summary and discussion about what is the Technological Singularity helped you understand the exciting, albeit dangerous times we are approaching.

From now on, the only thing we can be sure is Change.

And that such Change will happen at an exponential rate, so we must be capable of adapting swiftly to the new environment, both at the personal and business level.

But for those of us who have acquired the ability to turn those threats into opportunities, and absorbing the new technologies as they come, these will be times of success without precedent.

Welcome to the New World.

Thank you for reading my publication, and if you consider it helpful or inspiring, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

Hello everyone.

In my previous posts I’ve covered already:

• The Prohibited Artificial Intelligence Practices in the European Union1

• The definition of High Risk AI Systems in the European Union2

Now, we will discuss what are the Requirements for those AI Systems classified as High Risk by the European Union AI Act3.

High Risk AI Systems

The Chapter III: Section 2 is aptly named Requirements for high risk AI systems.

It’s first article clearly states that the requirements defined in this section are of mandatory compliance for the AI systems classified as High Risk, and subject to evaluation given the state of the art in AI, and their intended purpose.

Also it makes clear that the responsibility of ensuring compliance of the High Risk AI systems subject to this regulation are the providers of such systems.

And it is allowed, to avoid duplication of procedures, to integrate the risk management procedures into the ones that might already exist for the products who are already under oversight, listed in the Annex I of the EU AI Act.

Risk Management System

High Risk AI Systems are required to be monitored and managed through a Risk Management System.

It is defined as a continuous and iterative process that must be present during the whole lifecycle of the High Risk AI System, and must be updated if needed accordingly.

The Risk Management System must be targeted at the known and the reasonably foreseeable risks that the high-risk AI system can pose to health, safety or fundamental rights when the high-risk AI system is used in accordance with its intended purpose (taken from the Risk Management Section of the EU AI Act).

It follows the typical risk management process:

• Identify the risks, as stated previously

• Estimate and evaluate the risks that can emerge from usage for their intended purpose, and also reasonably foreseeable misuse

• Evaluate other risks that can come from the Post-Market Surveillance defined in the EU AI Act (Article 72)

• Adopt the appropriate and targeted risk management measures

According to the Article, the only risks that are of concern are those that can be reasonably eliminated or mitigated through development or technical information (this means not wasting effort and resources on things that cannot be managed).

Once all the risk management measures are taken into account, the remaining overall residual risk needs to be evaluated as acceptable to proceed.

To decide the best possible risk management measures, the following protocol is recommended:

• Elimination of risks as far as it is technically feasible, through design and development of the AI System

• Mitigation and Control measures for those risks that cannot be eliminated

• Training and Information for deployers, with the intent of eliminating or reducing risks under the presumable context where the AI system will be used

High Risk AI Systems should be tested throughout their development process, prior to being placed on the market, to identify the most appropriate risk management measures.

Such tests should ensure the AI systems perform consistently according to their intended purpose.

When real world testing is going to take place, it should do so in accordance with Article 60: Testing of High-Risk AI Systems in Real World Conditions Outside AI Regulatory Sandboxes.

The tests should be carried against prior defined metrics and thresholds, appropriate to the intended purpose.

This is to avoid AI developers trying to conform the tests to the product, instead of measuring the product against the required tests.

If the High Risk AI System is likely to have a negative impact on persons under the age of 18 or vulnerable populations, its risk management system must receive special consideration.

And finally, if the High Risk AI System is part of a product that is already subject to risk management processes under the Union’s law, the new risk management requirements can be combined into them, to avoid duplication.

Subscribe now

Share

Data and Data Governance

For High Risk AI Systems that involve the training of AI models, they will require that the data used complies with the quality criteria from this regulation.

Training, validation and testing data sets should be subject to data governance and management practices appropriate for the intended purpose of the High Risk AI System.

Citing the EU AI Act, those practices should be applied to:

• the relevant design choices;

• data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection;

• relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation;

• the formulation of assumptions, in particular with respect to the information that the data are supposed to measure and represent;

• an assessment of the availability, quantity and suitability of the data sets that are needed;

• examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations;

• appropriate measures to detect, prevent and mitigate possible biases identified;

• the identification of relevant data gaps or shortcomings that prevent compliance with this Regulation, and how those gaps and shortcomings can be addressed.

It is expected that the training, validation and testing data sets are relevant, sufficiently representative, free of errors and complete given their intended purpose.

They must possess the appropriate statistical properties, taking into consideration the population where the High Risk AI System is intended to be used.

This means that those data sets must be crafted accordingly to the specific geographical, contextual, behavioral or functional setting where the High Risk AI System is expected to operate.

To ensure bias detection and correction, it might be necessary to exceptionally process special categories of personal data, subject to appropriate safeguards for the fundamental rights and freedoms of natural persons.

According to the EU AI Act, in addition to the provisions set out in Regulations (EU) 2016/679 and (EU) 2018/1725 and Directive (EU) 2016/680, all the following conditions must be met in order for such processing to occur:

• the bias detection and correction cannot be effectively fulfilled by processing other data, including synthetic or anonymised data;

• the special categories of personal data are subject to technical limitations on the re-use of the personal data, and state-of-the-art security and privacy-preserving measures, including pseudonymisation;

• the special categories of personal data are subject to measures to ensure that the personal data processed are secured, protected, subject to suitable safeguards, including strict controls and documentation of the access, to avoid misuse and ensure that only authorised persons have access to those personal data with appropriate confidentiality obligations;

• the special categories of personal data are not to be transmitted, transferred or otherwise accessed by other parties;

• the special categories of personal data are deleted once the bias has been corrected or the personal data has reached the end of its retention period, whichever comes first;

• the records of processing activities pursuant to Regulations (EU) 2016/679 and (EU) 2018/1725 and Directive (EU) 2016/680 include the reasons why the processing of special categories of personal data was strictly necessary to detect and correct biases, and why that objective could not be achieved by processing other data.

And finally, High Risk AI Models that are developed without training AI models only need to comply the specified data governance requirements for the testing data sets.

Technical Documentation

The High Risk AI System’s technical documentation needs to be available before the system is placed onto the market or put into service and must be constantly updated.

The technical documentation must be created in a way that demonstrates the High Risk AI System complies with the requirements established on this regulation, so the competent authorities can assess compliance too.

SMEs and startups can provide the documentation defined in Annex IV: Technical Documentation in a simplified manner, that will be established by the Commission.

If the product is covered in the Annex I: List of Union Harmonization Legislation, then it should provide a unified documentation that includes what it is already enforced by the Union legislation and what is required by the EU AI Act.

It is important to note that the Commission is empowered to update the Technical Documentation requirements (Annex IV) as needed due to technical progress in the AI field.

Record Keeping

High Risk AI Systems must allow for automatic logging of events during their entire lifetime at the technical level.

The logged events should be relevant to:

• identifying situations that may result in the high-risk AI system presenting a risk within the meaning of Article 79(1): Risks of National Level or in a substantial modification;

• facilitating the post-market monitoring referred to in Article 72; and

• monitoring the operation of high-risk AI systems referred to in Article 26(5).

This means that the logging system should be capable of registering events that cover Risks of National Level (Article 79-1) detected during the Post-Market Surveillance (Article 72) by any actor during the AI operation, being especially mentioned the deployers (Article 26-5).

Such logging capabilities should provide, at a minimum:

• recording of the period of each use of the system (start date and time and end date and time of each use);

• the reference database against which input data has been checked by the system;

• the input data for which the search has led to a match;

• the identification of the natural persons involved in the verification of the results, as referred to in Article 14(5).

Subscribe now

Share

Transparency and Provision of Information to Deployers

High Risk AI Systems must be designed and developed in such a way as to ensure its operation is transparent enough to enable deployers to interpret its output, allowing compliance for both deployer and provider.

The instructions should be provided in an appropriate format, being complete, concise, correct, clear, relevant and accessible to the deployers.

Such instructions must contain at least the following information:

• the identity and the contact details of the provider and, where applicable, of its

  authorised representative;

• the characteristics, capabilities and limitations of performance of the high-risk AI system, including:

  • its intended purpose;

  • the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15: Accuracy, Robustness and Cybersecurity, against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity;

  • any known or foreseeable circumstance, related to the use of the high-risk AI system in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, which may lead to risks to the health and safety or fundamental rights referred to in Article 9(2): Risk Management System;

  • where applicable, the technical capabilities and characteristics of the high- risk AI system to provide information that is relevant to explain its output;

  • when appropriate, its performance regarding specific persons or groups of persons on which the system is intended to be used;

  • when appropriate, specifications for the input data, or any other relevant information in terms of the training, validation and testing data sets used, taking into account the intended purpose of the high-risk AI system;

  • where applicable, information to enable deployers to interpret the output of the high-risk AI system and use it appropriately;

• the changes to the high-risk AI system and its performance which have been pre- determined by the provider at the moment of the initial conformity assessment, if any;

• the human oversight measures referred to in Article 14: Human Oversight, including the technical measures put in place to facilitate the interpretation of the outputs of the high-risk AI systems by the deployers;

• the computational and hardware resources needed, the expected lifetime of the high-risk AI system and any necessary maintenance and care measures, including their frequency, to ensure the proper functioning of that AI system, including as regards software updates;

• where relevant, a description of the mechanisms included within the high-risk AI system that allows deployers to properly collect, store and interpret the logs in accordance with Article 12: Record Keeping.

Human Oversight

High Risk AI Systems must be subject to human oversight during their operation, and for that they must have been built with the proper interfaces.

The purpose of such oversight is to minimize the risks to health, safety or fundamental rights that might come from the AI system.

Such oversight measures must be commensurate with the risks, level of autonomy and context of use of the High Risk AI System, and ensured through one of the two following measures:

• when technically feasible, oversight measures should be built into the High Risk AI System before it is deployed on the market or put into service by the provider

• if not possible to integrate the oversight measures into the AI system before it is deployed, those measures should be implemented by the deployer.

The High Risk AI System should be provided to the deployer in such a way that the natural persons responsible of human oversight are enabled to:

• to properly understand the relevant capacities and limitations of the high-risk AI system and be able to duly monitor its operation, including in view of detecting and addressing anomalies, dysfunctions and unexpected performance

• to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons

• to correctly interpret the high-risk AI system’s output, taking into account, for example, the interpretation tools and methods available;

• to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system

• to intervene in the operation of the high-risk AI system or interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state

Special care is placed into the use case of AI-powered remote biometric identification, present on Annex III(1a).

The High Risk AI System, operated by the deployer, must not be able to validate the identification unless at least two different natural persons with the necessary competency, training and authority, have validated the result separately.

This requirement of separate verification will not apply to high-risk AI systems used for the purposes of law enforcement, migration, border control or asylum, where Union or national law considers the application of this requirement to be disproportionate.

Subscribe now

Share

Accuracy, Robustness and Cybersecurity

High Risk AI Systems must be designed and developed in such a way that they achieve an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently in those respects throughout their lifecycle.

Benchmarking and measurement methodologies will be developed by the Commission with the help of the national metrology and benchmarking authorities.

The levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use.

High Risk AI Systems must take into account the context where they’ll be deployed, and be resilient as possible to errors, especially due to interaction with persons.

To achieve robustness, redundancy, backups and failsafes should be considered as part of the technical solutions.

Another possible source of problems can come from High Risk AI Systems that continue learning after being put on the market.

Feedback loops of biased outputs influencing future inputs should be eliminated or mitigated appropriately.

High-risk AI systems shall be resilient against attempts by unauthorized third parties to alter their use, outputs or performance by exploiting system vulnerabilities.

The type of attacks considered should include, but not be limited to:

• attacks trying to manipulate the training data set (data poisoning)

• pre-trained components used in training (model poisoning)

• inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion)

• confidentiality attacks or model flaws

Conclusion

High Risk AI Systems have a very strict set of requirements they need to be compliant with over their whole lifecycle: from development to deployment and operation, until they are finally retired.

Appointed authorities and regulatory bodies are responsible of collecting the compliance information.

But it is the providers and deployers who in the end share the responsibility over the proper operation and compliance of the High Risk AI Systems under their control.

We’ll review that topic in my next article.

See you in the next one!

Thank you for reading my publication, and if you consider it helpful or inspiring, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

In my previous article “Prohibited Artificial Intelligence Practices in the European Union1” we discussed the explicitly prohibited practices regarding the use of AI systems in the EU.

But once we take all of the use cases that are forbidden, we still have those AI Systems that are considered dangerous enough for the wellbeing of the society, so they require special oversight.

Those have the denomination of High Risk AI Systems, and are subject to a series of strict requirements and registration in an ad-hoc EU High Risk AI Systems Database for surveillance once they are in the market.

Classification of AI Systems as High Risk

The first step is to clearly identify which AI Systems are considered as High Risk, so they are covered by the EU regulation.

There are two sources of High Risk AI Systems defined in the EU AI Act:

• The European Union Harmonization Legislation, listed on the Annex I from the EU AI Act2

• The list of AI Systems provided on the Annex III from the EU AI Act

The Annex I has a list of regulated products and systems, which because of their critical nature are subject to special scrutiny, even without considering AI itself.

The EU AI Act considers that every product and system in that list can be considered a High Risk AI System, if it is implemented using AI as the product itself, or if it uses AI as part of its safety systems.

We will review them here.

High Risk AI Systems from the Harmonization Legislation (Annex I)

This is a complete, but simplified list of the Harmonization Legislation list in the Annex I from the EU AI Act.

• Directive of Machinery Regulation

• Directive on the Safety of Toys

• Directive on Recreational Craft and Personal Watercraft

• Directive on Lifts and Safety Components for Lifts

• Directive on Equipment and Protective Systems intended for use on Potentially Explosive Atmospheres

• Directive on the market of Radio Equipment

• Regulation on Cableway installations

• Regulation on Protective Equipment

• Regulation on Appliances burning Gaseous Fuels

• Regulation on Medical Devices

• Regulation on In Vitro Diagnostic Medical Devices

• Regulation on Common Rules in the field of Civil Aviation Security

• Regulation on the market surveillance of two and three wheel vehicles and quadricycles

• Regulation on the approval and market surveillance of Agricultural and Forestry vehicles

• Directive on Marine Equipment

• Directive on the interoperability of the Rail System within the European Union

• Regulation on the approval and market surveillance of motor vehicles and their trailers, and of systems, components and separate technical units intended for such vehicles

• Regulation on type-approval requirements for motor vehicles and their trailers, and systems, components and separate technical units intended for such vehicles, as regards their general safety and the protection of vehicle occupants and vulnerable road users

• Regulation on common rules in the field of civil aviation and establishing a European Union Aviation Safety Agency

For the EU AI Act, if an AI system is intended to be used as a safety component for any of the product categories cited before, it is automatically considered a High Risk AI System, for the purposes of this regulation.

It also applies if the AI system itself is the product, and it falls within the categories here mentioned.

For example, if an AI system is designed to perform as part of the safety system of a lift, it is considered a High Risk AI System, and must comply with all the requirements stated by the EU AI Act.

Or if an AI system’s purpose is to control the operation of an appliance that burns gas (for efficiency gains, for example), it is also considered a High Risk AI System.

For greater detail on each product class regulated here, check the EU AI Act Annex I, where you will find the reference to the European Union normative that drives it.

High Risk AI Systems from the Annex III of the EU AI Act

These types of AI Systems will be considered of High Risk, and subject to the EU AI Act regulations and requirements here reviewed.

• Biometrics, if their use is allowed by the national laws of the Member State

  • Remote Biometric Identification Systems (as long as they are not solely used to verify and confirm a person is who he claims to be)

  • AI Systems intended to be used for biometric categorization

  • AI Systems intended to be used for emotion recognition

This is one of the most specifically regulated AI use cases in the EU AI Act, as we have explored in my previous article “Prohibited Artificial Intelligence Practices in the European Union”.

Also, the use of AI for Emotion Recognition is regulated only for the workplace and educational institutions, but freely allowed in other contexts (check my article3 on this topic).

• Critical infrastructure

  • AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic, or in the supply of water, gas, heating or electricity.

This is probably the most obvious one. Critical infrastructure can be the target of cyberattacks, and malfunctions in these systems can lead to severe harm or death for large portions of the population.

Subscribe now

Share

• Education and vocational training

  • AI systems intended to be used to determine access or admission or to assign natural persons to educational and vocational training institutions at all levels;

  • AI systems intended to be used to evaluate learning outcomes, including when those outcomes are used to steer the learning process of natural persons in educational and vocational training institutions at all levels;

  • AI systems intended to be used for the purpose of assessing the appropriate level of education that an individual will receive or will be able to access, in the context of or within educational and vocational training institutions;

  • AI systems intended to be used for monitoring and detecting prohibited behavior of students during tests in the context of or within educational and vocational training institutions.

One of the most promising use cases of AI resides in educational institutions, and we already have many platforms to show.

Nevertheless, this is a very delicate context of application, as it can have a profound negative impact in society if not performed carefully.

AI systems are already being used as a tool for teachers to help assess large amounts of students homework.

Hallucinations and bad AI performance can lead to improper evaluations and grades, that in some cases will lead to additional manual reviewing, and in the worst case, to student failures.

And if AI is used to assess the appropriateness of students for certain careers, courses, or vocational paths, it can lead to missed opportunities that will harm students and society in the long term.

• Employment, workers management and access to self-employment

  • AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates;

  • AI systems intended to be used to make decisions affecting terms of work-related relationships, the promotion or termination of work-related contractual relationships, to allocate tasks based on individual behavior or personal traits or characteristics or to monitor and evaluate the performance and behavior of persons in such relationships.

The access to employment opportunities can also be negatively affected by biased or badly trained AI.

We already have AI systems sifting through thousands of job applications, without the intervention of humans in the process.

This has affected the way people craft their resumes, as it is necessary now to assume it won’t be read by a human, but by an AI.

So, techniques have been developed to fool AI systems into giving priority to some resumes over others.

Also job applicants are starting to use AI systems to apply massively to job offers, automating most of the process and trying to trick the HR AI systems into giving them priority.

It’s an arms race.

• Access to and enjoyment of essential private services and essential public services and benefits

  • AI systems intended to be used by public authorities or on behalf of public authorities to evaluate the eligibility of natural persons for essential public assistance benefits and services, including healthcare services, as well as to grant, reduce, revoke, or reclaim such benefits and services;

  • AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems used for the purpose of detecting financial fraud;

  • AI systems intended to be used for risk assessment and pricing in relation to natural persons in the case of life and health insurance;

  • AI systems intended to evaluate and classify emergency calls by natural persons or to be used to dispatch, or to establish priority in the dispatching of, emergency first response services, including by police, firefighters and medical aid, as well as of emergency healthcare patient triage systems;

If governments start using AI systems to assess eligibility for social services, perform triage in emergency situations, etc. then proper measures to avoid biases must be in place.

Also, if the private sector uses AI to evaluate individuals to decide if they should provide them services or not, those systems must be proven fair and unbiased.

Imagine a situation where your access to financial services, healthcare, or insurance is subject to the output of an AI system that has been trained with biases that affect you.

Subscribe now

Share

• Law enforcement, as long as its use is permitted by the Union or national laws

  • AI systems intended to be used by or on behalf of law enforcement authorities, or by Union institutions, bodies, offices or agencies in support of law enforcement authorities or on their behalf to assess a natural person’s risk of becoming the victim of criminal offences;

  • AI systems intended to be used by or on behalf of law enforcement authorities or by Union institutions, bodies, offices or agencies in support of law enforcement authorities as polygraphs or similar tools;

  • AI systems intended to be used by or on behalf of law enforcement authorities, or by Union institutions, bodies, offices or agencies, in support of law enforcement authorities to evaluate the reliability of evidence in the course of the investigation or prosecution of criminal offences;

  • AI systems intended to be used by law enforcement authorities or on their behalf or by Union institutions, bodies, offices or agencies in support of law enforcement authorities for assessing the likelihood of a natural person of offending or re- offending not solely based on profiling of natural persons as referred to in Article 3(4) of Directive (EU) 2016/680, or to assess personality traits and characteristics or past criminal behaviour of natural persons or groups;

  • AI systems intended to be used by or on behalf of law enforcement authorities or by Union institutions, bodies, offices or agencies in support of law enforcement authorities for the profiling of natural persons as referred to in Article 3(4) of Directive (EU) 2016/680 in the course of the detection, investigation or prosecution of criminal offenses.

The use of AI systems for law enforcement is the most regulated AI use case in the EU AI Act, in the form of Realtime Biometric Identification Systems.

The risk of employing biased or badly trained AI models in law enforcement can directly impact the fundamental rights of the population.

Profiling systems based on biometric and personal characteristics, face or voice recognition in public spaces, criminal offense prediction systems, etc. are all strictly regulated.

In particular, biometric identification systems must be approved by a judicial authority before being used (like current wiretapping systems).

Also, the AI systems employed must be registered within the EU High Risk AI Systems Database, and every use must be reported to an appointed authority, to be later reported to an EU Commission.

The aggregate results of the use of such law enforcement AI systems will be made public yearly by the EU agency responsible.

If you are interested in the details, check my previous article: “Prohibited Artificial Intelligence Practices in the European Union”.

• Migration, asylum and border control management, as long as its use is permitted by the Union or national laws

  • AI systems intended to be used by competent public authorities as polygraphs and similar tools;

  • AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies to assess a risk, including a security risk, a risk of irregular migration, or a health risk, posed by a natural person who intends to enter or who has entered into the territory of a Member State;

  • AI systems intended to be used by or on behalf of competent public authorities or by Union institutions, bodies, offices or agencies to assist competent public authorities for the examination of applications for asylum, visa or residence permits and for associated complaints with regard to the eligibility of the natural persons applying for a status, including related assessments of the reliability of evidence;

  • AI systems intended to be used by or on behalf of competent public authorities, including Union institutions, bodies, offices or agencies, in the context of migration, asylum or border control management, for the purpose of detecting, recognising or identifying natural persons, with the exception of the verification of travel documents.

Every use case where large amounts of data needs to be processed and vetted can potentially benefit from using an AI system.

But as with the previously reviewed scenarios, if the assessment can be afflicted by biases introduced either by data during training, or biased algorithms during development, those systems require special oversight.

Especially if an improper output can impact fundamental rights and freedoms from people, like in the case of migration eligibility assessments.

• Administration of justice and democratic processes

  • AI systems intended to be used by a judicial authority or on their behalf to assist a judicial authority in researching and interpreting facts and the law and in applying the law to a concrete set of facts, or to be used in a similar way in alternative dispute resolution;

  • AI systems intended to be used for influencing the outcome of an election or referendum or the voting behaviour of natural persons in the exercise of their vote in elections or referenda. This does not include AI systems to the output of which natural persons are not directly exposed, such as tools used to organize, optimize or structure political campaigns from an administrative or logistical point of view.

The sheer amount of content that represents the legislation and regulations from a country, or worse, a bloc like the European Union, makes it a prime use case for AI systems.

AI systems trained in the laws, cases and doctrine from the corresponding jurisdictions where they are intended to be used can be of great help in speeding up the administration of justice.

Not only by facilitating the finding of obscure or little known related information, but also by providing summaries and even some level of reasoning and interpretation.

And, as with the other scenarios already reviewed, biases and improper training can lead to severe harm of the wide population.

The same can happen in the use of AI on democratic processes. We have already witnessed the harmful results of this in the Cambridge Analytica scandal.

Subscribe now

Share

Exceptions and Amendments

Of course, for every rule there are always exceptions.

In this case, AI systems that could have been classified as High Risk for the purposes of the EU AI Act can be exempted if they do not pose a significant risk of harm to the health, safety or fundamental rights of natural persons.

Also are exempted the use cases that do not directly influence the outcome of a decision process.

This is the exhaustive list:

• the AI system is intended to perform a narrow procedural task;

• the AI system is intended to improve the result of a previously completed human activity;

• the AI system is intended to detect decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment, without proper human review; or

• the AI system is intended to perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III.

This exemption enables the use of AI systems as a tool to enhance human processes, with specific objectives and always under final human oversight.

Regardless of this, any AI system that conducts profiling of natural persons will be considered as High Risk.

If the developer of an AI system classifiable as High Risk considers its special case doesn’t belong to this classification, he can apply for an exemption at the proper EU regulatory body, by presenting an assessment of his reasons.

And finally, the EU Commission will be permanently reassessing the High Risk AI Systems list, to add or modify its use cases.

So we need to check that list periodically, if we are involved either in the development or adoption of such AI systems.

Conclusion

We have reviewed the use cases that are considered by the European Union Artificial Intelligence Act as High Risk AI Systems.

Most of them were already heavily regulated for other reasons, as consumer safety, national security, etc.

As entrepreneurs building products that might be using AI as part of it, organizations considering acquiring and deploying an AI system, or the general public, we need to be aware of which AI use cases are considered as High Risk.

Now that we have identified the High Risk AI System use cases, we will discuss in my next article what are the requirements that such systems must comply to be able to operate in the EU market.

Thank you for reading my publication, and if you consider it helpful or inspiring, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

A couple of days ago, I watched a tech influencer’s reel celebrating how the Instituto Tecnológico de Aragón (Aragón’s Institute of Technology, Spain) had succeeded in using AI to analyze the emotional response of people at a dining table to the food and service they received1.

It immediately brought to mind how potentially dangerous things can turn out if we leave researchers, governments and for-profit organizations to their own devices and appetites in everything related to the use of AI.

It wasn’t long before media outlets started praising the concept. El Heraldo (Spain) explained how this AI system could be used to analyze in real-time the response of diners to the replacement of more expensive ingredients with cheaper ones2.

While La Vanguardia (Spain) makes noteworthy the idea of concealing the required 360º camera in a vase-like device in the middle of the table to collect the diners’ reactions while they eat3.

El País (Spain) participated in one of the experiments, taking very interesting pictures and opinions from the researchers themselves4.

What if I don’t want to opt in?

In March of this year, the European Union finally passed its Artificial Intelligence Act, and within it, strict regulations and definitions of Prohibited AI Practices within their jurisdiction (check my recent article on that topic)5.

In its Chapter II: Prohibited Artificial Intelligence Practices, the EU AI Act explicitly forbids the use of AI to infer emotions in the workplace and education institutions.

the placing on the market, the putting into service for this specific purpose, or the use of AI systems to infer emotions of a natural person in the areas of workplace and education institutions, except where the use of the AI system is intended to be put in place or into the market for medical or safety reasons.

But it says nothing about the use of AI systems aimed to infer emotional responses in public areas like restaurants and commerces, with the intent of marketing and product research, without the consent of the people involved.

The developers of an AI system created with such purpose only need to register it within the European Union High Risk AI Systems Database and obtain the proper certification to market it in the EU jurisdiction.

So, in theory, the AI use case explored by the Instituto Tecnológico de Aragón is valid, and it doesn’t require an explicit authorization from the diners involved.

A peek into our (potential) future

This situation reminds me of an interesting situation proposed by the sci-fi series, Babylon 5.

In the future depicted by Babylon 5, negotiations of all sorts (commercial, government, private, etc.) were allowed to happen with the presence of a “Commercial Telepath”6.

Commercial Telepaths were there to validate if the parts in a negotiation were being truthful, by directly reading their minds during the process.

It was a government sanctioned and regulated profession, and it was seen as the natural way to do things.

Subscribe now

Share

Now we live in an era where we might be subject to constant analysis by AI-enabled tools that will give an additional edge to businesses, governments and organizations.

Power asymmetry increases in favor of organizations, and in detriment of the individuals.

Right when we are fighting hard to recover our right to privacy while browsing the Internet, by stopping the all-pervasive marketing trackers placed into us by Facebook, Google and others.

The Road Ahead

I’m a tech enthusiast, as any good Software Engineer should be. I’ve been involved in AI stuff for at least 25 years. And I think it will be an important part of our future.

But it is our responsibility, as individuals, as responsible citizens, to keep things under control.

The Lifecycle of AI systems involve those with some AI-based idea, trying to make it happen, those who are going to acquire and deploy the AI system for their commercial benefit, and those who are going to be affected by the use of such AI.

It doesn’t matter where we are located in the AI market spectrum (creators, deployers, users, or regulators) we have to keep ourselves aware and demand that every advancement happens in our benefit.

This might mean proposing that the protections the AI regulations entail also cover commercial situations like the ones depicted in this post.

What do yo think about this?

Let me know in the comments below!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

Last March 13th, the European Parliament finally voted and passed the much debated European Union Artificial Intelligence Act1. And within it, the detailed definition of the Prohibited Artificial Intelligence Practices in the European Union, which we all should be aware of, regardless of our position in the market: entrepreneurs building Artificial Intelligence tools within the European Union jurisdiction, entrepreneurs building AI tools outside of the EU, but with the intent of selling it within the EU jurisdiction, organizations within the EU (private and public) intending to acquire, develop, or deploy AI tools, users of AI tools within such organizations, or the the general public which will be affected by the use of AI. Everyone.

The road hasn’t been short nor easy. Preceded by the EU Digital Markets Act2 and strongly based on the output from the EU High Level Expert Group on Artificial Intelligence’s Ethics Guidelines for Trustworthy AI3, it has been subject to a lot of debate. At some point, a few of the most actively engaged EU Member States were even considering withdrawing from the Act itself4, because they considered it too restrictive and with the potential of hampering their AI development.

I’ve written a few pieces on that regard if you wanna check them out:

• Trustworthy Artificial Intelligence - Part 1

• Trustworthy Artificial Intelligence - Part 2

• Trustworthy Artificial Intelligence - Part 3

• Trustworthy Artificial Intelligence - Part 4

The European Union Artificial Intelligence Act is quite large, so we will deal with it in several parts, focusing on the most relevant sections of it, and leaving out most of the obscure legalese that is meant to give it structure or connect it with previous regulations (without ignoring their effects, of course).

We could split its areas of interest like this:

• Prohibited Artificial Intelligence Practices (the subject of this article)

• Definition of High Risk AI Systems, and How every party should deal with them

• Transparency Obligations for Providers and Deployers of AI Systems

• Obligations for the Providers of General Purpose AI Models

• Measures in Support of Innovation (or How do we plan to regulate all of this without scaring away from the EU all the AI researchers and entrepreneurs)

• Governance and the EU Database for High Risk AI Systems

• Post-Market Monitoring, Market Surveillance and Penalties

Let’s get on with it.

General Provisions

Let’s begin with the General Provisions, as stated on the EU AI Act Chapter 1:

“The purpose of this Regulation is to improve the functioning of the internal market and promote the uptake of human-centric and trustworthy artificial intelligence (AI), while ensuring a high level of protection of health, safety, fundamental rights enshrined in the Charter of Fundamental Rights, including democracy, the rule of law and environmental protection, against the harmful effects of artificial intelligence systems (AI systems) in the Union, and to support innovation.”

As mentioned before, the EU AI Act is highly influenced by the previous work of the Trustworthy AI Guidelines, developed by the EU High Level Expert Group on AI. It starts with the Fundamental Rights and moves from there into the more applied objectives of trying to balance the containment of potential harm produced by AI systems, with the necessity of supporting innovation on this field.

Who are the market actors affected by the scope of this AI regulation?

• Providers of AI Systems located both in and outside of the EU

• Deployers of AI Systems located in the EU

• Providers and Deployers outside of the EU, but the AI System’s output is used inside the EU

• Importers and Distributors of AI Systems

• Product manufacturers that embed AI Systems within their products

• Authorized representatives of providers not established in the EU

• Affected persons located in the EU

Which actors are outside the scope of the EU AI Act?

Outside of the scope are the National Security concerns of the Member States, and military applications. Also aren’t affected states not member of the EU who make use of their own AI systems during cooperations, as long as they can provide proof that such systems protect the fundamental rights and freedoms as stated by the EU charter.

AI Systems and AI Models in operation for scientific purposes, research and development are outside of the scope of the Act, until they enter real world testing phase. At that point they are liable to the current regulation.

Consumer protection and product safety laws apply normally to everything AI related.

Individuals tinkering with AI outside of any professional activity are not subject to the Act. So yeah, you can build Skynet in your basement, as long as you don’t openly sell it, or test it in the public domain.

The Member States can introduce further regulations to favor workers affected by AI Systems.

AI Systems developed under Free and Open Source licenses are not subject to the AI Act, unless they are placed on the market or are deemed High Risk AI Systems, considered as Prohibited AI Practices, or should be subject to the Transparency Obligations cited in Article 50.

Subscribe now

Share

Prohibited Artificial Intelligence Practices

This list can be quite extensive, so we will review its elements to assess their impact for both end users and entrepreneurs engaging in AI development.

I’ve simplified a bit the legalese to make it more readable, but if you want to check the original text (in the EU AI Act Chapter 2), you can refer to the source material in the References section.

• AI systems that use subliminal techniques beyond a person’s or group of persons consciousness to purposely manipulate their behavior, leading them to taking decisions they would have not otherwise taken and that will result in significant harm to them.

• AI systems that exploit vulnerabilities of a person or group of persons (like age, disability, social or economic situation) to manipulate them and lead them to take decisions that will result in significant harm.

The EU AI Act is very conscious of the dangers AI can pose to the general population in terms of manipulating the human behavior through targeted and repeated bombardment of information. Most probably building upon the negative experiences from the Cambridge Analytica scenario, and related ones, it explicitly forbids the use of AI for this use case.

• AI systems deployed with the intention of evaluating or classifying persons according to their behavior, inferring personal characteristics, storing that social score and using it to treat them unfavorably in a different context, or in a disproportionate way given the behavior analyzed.

• AI systems deployed with the intention of making risk assessments of persons to predict the likelihood of him committing a criminal offense, based solely on the profiling performed on their personal traits or characteristics.

• AI systems can be used as a tool to support the assessment being performed by a human on the likelihood of a person having participated in a criminal activity, based on existing and verifiable facts linked to such activity.

AI systems aren’t new. We already have real examples of AI systems used to classify people for law enforcement purposes, resulting in inadequate and even racist outputs, as reported by this MIT Technology Review article5.

We also have the Police State being built by China and its famous “Social Scoring System”, already in operation.

Building upon these experiences, the EU AI Act also explicitly forbids the use of AI systems to perform classification and inference of future behavior of people.

Nevertheless, the use of AI systems as a tool to sift over large amounts of data to find or refine information a human criminal investigator is already working on is also explicitly permitted.

• AI systems with the purpose of creating or expanding a facial recognition database by scraping public, CCTV or Internet video sources.

• AI systems aimed to infer the emotions of natural persons in the workplace or educational institutions, except for medical or safety reasons.

• AI systems with the intent of categorizing and classifying people according their sex, race, political opinion, beliefs, etc. deducing them from their biometric data.

The potential of using AI systems to scrape biometric information from public sources is one of the biggest risks this technology poses, and it is being addressed directly here.

Also, the use of AI to infer the emotional state of people can be used for manipulation (think about emotion-based advertising, for example). This is also explicitly forbidden.

And any attempts to classify people, by inferring their personal characteristics from their interactions with any system (for example, Social Media) is considered a forbidden practice.

Subscribe now

Share

• It is permitted the use of AI systems to perform labelling or filtering of lawfully acquired biometric data in the area of law enforcement.

• The use of real-time biometric identification systems in public spaces for the purposes of law enforcement, unless it is deemed strictly necessary to achieve one of the following objectives

  • The targeted search of victims of abduction, human trafficking, sex exploitation, and searching for missing persons.

  • The prevention of a specific, substantial and imminent threat to the life and physical safety of persons, or a genuine threat of terrorist attack.

  • The localization or identification of a person suspected of having committed a criminal offense of the types referred at the Annex II.

• The use of real-time biometric identification in publicly accessible spaces for the purposes of law enforcement shall only be deployed to confirm the identity of the specific targeted individual and must take into account:

  • The scale and probability of the harm that would come if not used.

  • The consequences for the rights and freedoms of the people involved.

Another important use case of AI systems that needed regulation was the use of real-time biometric identification in public and open spaces, for law enforcement purposes.

This use case of AI is probably the most specifically regulated in the Forbidden Practices section of the EU AI Act.

It can only be approved in very specific cases, as stipulated in the Annex II: List of Criminal Offenses, from the Act, and those situations detailed here.

Nevertheless, AI systems are permitted as tools to help in the manipulation of data already acquired in a lawful manner, such as filtering and labelling.

• The use of real-time biometric identification systems in public spaces for the purposes of law enforcement can only be authorized only if such law enforcement authority has completed a Fundamental Rights assessment and the system to be used is registered in the EU High Risk AI Systems database.

• The authorization will be provided by either by a judicial authority or a specially appointed independent administrative authority whose decision is binding of the Member State.

• In cases of justified urgency, such systems could be used without an existing registration, but the registration process must be started within 24 hours at most, and if the authorization is rejected, the use of the AI system must be stopped immediately, and the information collected must be destroyed, as it cannot be used lawfully.

• The authorization provided is only valid for a specific geographic location, time, and personal scope.

• No decision that produces an adverse legal effect on a person can be taken by using solely the output of the real-time biometric identification system.

• Each and every use of the real-time biometric identification system must be notified to the Market Surveillance Authority and the National Data Protection Authority.

The EU AI Act has put a lot of effort in trying to control the governments’ ability to perform automated surveillance over their citizens.

This is clearly shown by the strict requirements in place to even use an AI system for real-time identification in public spaces.

Not only those systems must be registered previously to their usage, but each and every use must be directly approved by a judiciary authority, or a similar legally binding one appointed by their respective government, akin to the lawfully authorized wiretapping and surveillance of communications.

But in addition to that, data collected this way must be reported to a central EU authority, responsible of summarizing this information for later public reporting, and it cannot be the sole evidence to be used in a legal proceedings.

Subscribe now

Share

• Member States who decide to adopt real-time biometric identification systems must modify their national laws to allow it, in strict compliance of the limits and conditions already mentioned. After doing so, they must notify the EU Commission 30 days after, at the latest.

• Each national Market Surveillance Authorities and Data Protection Authorities will collect the notifications received, and report them annually to the EU Commission.

• The EU Commission shall publish annual reports on the consolidated data of the use of real-time biometric identification systems.

And finally, the Member States who wish to enable the use of AI systems capable of real-time biometric identification must explicitly modify their national laws, making them as restrictive as the EU AI Act or more, but not less.

They must also create the appropriate national agencies responsible of monitoring the compliance and data collection already mentioned, because there is an obligation of publishing consolidated reports on the topic, for general scrutiny.

Conclusion

This is the first article in the series where we’ll explore the most relevant elements of the EU AI Act for the general public and entrepreneurs wishing to create AI systems and market those in the European Union.

The efforts to regulate AI all over the world are just starting: the US Government has also issued Executive Orders on this topic (which I will also review in later articles), and we can bet that very soon additional jurisdictions will start aligning their legal frameworks to both reference points.

As individuals, as entrepreneurs, and as responsible citizens we must keep ourselves informed and aware of these regulations, because the usage of AI systems will deeply change the way we interact with each other, with our governments, and potentially tilt the power structures not necessarily in our favor.

In the next article we will review the next important point in the EU AI Act: “Definition of High Risk AI Systems, and How every party should deal with them”, so stay tuned.

Thank you for reading my publication, and if you consider it helpful or inspiring, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

This is a four part series. This one is the fourth (and last) part of the series. You can find the others here:

• Trustworthy Artificial Intelligence (Part 1)

• Trustworthy Artificial Intelligence (Part 2)

• Trustworthy Artificial Intelligence (Part 3)

• Trustworthy Artificial Intelligence (Part 4)

So far we have been following the European Union Guidelines for Ethical and Trustworthy AI1, as proposed by the appointed High Level Expert Group on this subject by the EU Parliament.

We have reviewed the concepts of Trustworthiness when applied to software systems, and in particular to Artificial Intelligence systems, capable of taking decisions in our behalf.

That effort took us through the analysis of the Fundamental Rights and Ethical Principles that should guide the full lifecycle of AI: its inception, design, development, deployment, operation and maintenance, even when something isn’t explicitly covered by laws and regulations.

Then we reviewed the proposed model for managing the Requirements through the AI lifecycle, derived from the Fundamental Rights and Ethical Principles previously mentioned, and should be continuously evaluated.

To implement those requirements, organizations and governments must make use of both Technical and Non-Technical methods available, and permanently scan the industry for new advancements and risks that will require taking action.

Now we will review the proposed methods of assessment that should be used as a baseline to ensure the requirements have been addressed properly during the full lifecycle of the AI implementation.

It’s worth noting that the assessment checklists do not cover the first component of Trustworthy AI: its compliance with laws and regulations. Those should be evaluated according to the jurisdiction where such AI are deployed, and where they are intended to be used.

Also it is important to keep in mind that during the application of AI to the real world it might happen that some situation is not covered by any law, regulation, procedure or even one of the Requirements already stated.

Is in those situations where the Fundamental Rights and Ethical Principles should help us to decide the right course of action.

The AI Lifecycle and the Assessment Process

The process of creating an AI tool, deploying it into an organization, overseeing its operation, or maintaining/upgrading it over time requires the involvement of different types of stakeholders: government regulators, creators, business executives, business personnel, end users and the general community.

AI Governance

The organization can use its own Change Management processes to handle the inclusion of AI, by adding the assessment processes to it, or create a new ad-hoc one if there isn’t one in place already.

The different organizational actors have different but very important roles. We will list them here.

Management and Board

The organization can either develop internally or procure externally its AI systems, and those impacted by the introduction of AI should participate in the process, under the leadership of top management.

Compliance/Legal Department/Corporate Responsibility Department

These departments are responsible of monitoring the proper use of the Assessment List, and the modification of the organization’s policies on AI systems, evolving them over time to match the legal requirements of the jurisdiction where it operates.

Product and Service Development or equivalent

These departments use the Assessment List to evaluate the AI products being acquired or developed, and report the results to top management.

Quality Assurance

The Quality Assurance department uses the Assessment List as guidance to evaluate the AI products, collect the quality requirements from every department, contrasting them against the results obtained.

HR

The HR department should use the Assessment List as a guidance to ensure the diversity of the AI team, if the AI systems are being developed in-house, and also make sure the organization develops the proper AI competences to be able to operate and extract all the benefits possible from the AI systems being deployed.

Procurement

The Procurement department is responsible of making sure the AI systems being acquired by the organization are evaluated and comply with the Assessment List.

Day-to-Day Operations

The project managers, developers and personnel should include the Assessment List as part of their daily work.

Subscribe now

Share

Using the Trustworthy AI Assessment List

As mentioned before, this Assessment List is a proposed guideline on how to achieve Trustworthy AI in our organizations, government and society.

It can (and should) be enriched with the experience and processes already existing where it is going to be applied.

But also bear in mind that not every situation can be mapped to the Assessment List.

The proper course of action will be to use the Fundamental Rights and Ethical Principles to help us solve those situations, and exert judgement based on a diverse set of opinions when conflicts arise.

Relation to Law and Processes

The Assessment List does not guarantee compliance with the local laws, nor the laws of every jurisdiction where the AI system is intended to be used.

However, compliance with the Assessment List can, in some situations, overlap with the legal requirements already in place, or make easier to achieve legal compliance with some extra steps.

So, the Assessment List should be taken as one of the many steps necessary to achieve Trustworthy AI, and not as a one-stop solution.

Trustworthy AI Assessment List (Pilot Version)

Here we detail the Trustworthy AI Assessment List (the pilot version published by the EU Expert Group) that we can use to start preparing our organizations for Trustworthy AI compliance.

1. Human Agency and Oversight

Fundamental Rights

• Did you carry out a fundamental rights impact assessment where there could be a negative impact on fundamental rights? Did you identify and document potential trade-offs made between the different principles and rights?

• Does the AI system interact with decisions by human (end) users (e.g. recommended actions or decisions to take, presenting of options)?

  • Could the AI system affect human autonomy by interfering with the (end) user’s decision-making process in an unintended way?

  • Did you consider whether the AI system should communicate to (end) users that a decision, content, advice or outcome is the result of an algorithmic decision?

  • In case of a chat bot or other conversational system, are the human end users made aware that they are interacting with a non-human agent?

Human Agency

• Is the AI system implemented in work and labour process? If so, did you consider the task allocation between the AI system and humans for meaningful interactions and appropriate human oversight and control?

  • Does the AI system enhance or augment human capabilities?

  • Did you take safeguards to prevent overconfidence in or over-reliance on the AI system for work processes?

Human Oversight

• Did you consider the appropriate level of human control for the particular AI system and use case?

  • Can you describe the level of human control or involvement?

  • Who is the “human in control” and what are the moments or tools for human intervention?

  • Did you put in place mechanisms and measures to ensure human control or oversight?

  • Did you take any measures to enable audit and to remedy issues related to governing AI autonomy?

• Is there is a self-learning or autonomous AI system or use case? If so, did you put in place more specific mechanisms of control and oversight?

  • Which detection and response mechanisms did you establish to assess whether something could go wrong?

  • Did you ensure a stop button or procedure to safely abort an operation where needed? Does this procedure abort the process entirely, in part, or delegate control to a human?

2. Technical Robustness and Safety

Resilience to Attack and Security

• Did you assess potential forms of attacks to which the AI system could be vulnerable?

  • Did you consider different types and natures of vulnerabilities, such as data pollution, physical infrastructure, cyber-attacks?

• Did you put measures or systems in place to ensure the integrity and resilience of the AI system against potential attacks?

• Did you verify how your system behaves in unexpected situations and environments?

• Did you consider to what degree your system could be dual-use? If so, did you take suitable preventative measures against this case (including for instance not publishing the research or deploying the system)?

Fallback Plan and General Safety

• Did you ensure that your system has a sufficient fallback plan if it encounters adversarial attacks or other unexpected situations (for example technical switching procedures or asking for a human operator before proceeding)?

• Did you consider the level of risk raised by the AI system in this specific use case?

  • Did you put any process in place to measure and assess risks and safety?

  • Did you provide the necessary information in case of a risk for human physical integrity?

  • Did you consider an insurance policy to deal with potential damage from the AI system?

  • Did you identify potential safety risks of (other) foreseeable uses of the technology, including accidental or malicious misuse? Is there a plan to mitigate or manage these risks?

• Did you assess whether there is a probable chance that the AI system may cause damage or harm to users or third parties? Did you assess the likelihood, potential damage, impacted audience and severity?

  • Did you consider the liability and consumer protection rules, and take them into account?

  • Did you consider the potential impact or safety risk to the environment or to animals?

  • Did your risk analysis include whether security or network problems such as cybersecurity hazards could pose safety risks or damage due to unintentional behavior of the AI system?

• Did you estimate the likely impact of a failure of your AI system when it provides wrong results, becomes unavailable, or provides societally unacceptable results (for example discrimination)?

  • Did you define thresholds and did you put governance procedures in place to trigger alternative/fallback plans?

  • Did you define and test fallback plans?

Subscribe now

Share

Accuracy

• Did you assess what level and definition of accuracy would be required in the context of the AI system and use case?

  • Did you assess how accuracy is measured and assured?

  • Did you put in place measures to ensure that the data used is comprehensive and up to date?

  • Did you put in place measures in place to assess whether there is a need for additional data, for example to improve accuracy or to eliminate bias?

• Did you verify what harm would be caused if the AI system makes inaccurate predictions?

• Did you put in place ways to measure whether your system is making an unacceptable amount of inaccurate predictions?

• Did you put in place a series of steps to increase the system's accuracy?

Reliability and Reproducibility

• Did you put in place a strategy to monitor and test if the AI system is meeting the goals, purposes and intended applications?

  • Did you test whether specific contexts or particular conditions need to be taken into account to ensure reproducibility?

  • Did you put in place verification methods to measure and ensure different aspects of the system's reliability and reproducibility?

  • Did you put in place processes to describe when an AI system fails in certain types of settings?

  • Did you clearly document and operationalize these processes for the testing and verification of the reliability of AI systems?

  • Did you establish mechanisms of communication to assure (end-)users of the system’s reliability?

3. Privacy and Data Governance

Respect for Privacy and Data Protection

• Depending on the use case, did you establish a mechanism allowing others to flag issues related to privacy or data protection in the AI system’s processes of data collection (for training and operation) and data processing?

• Did you assess the type and scope of data in your data sets (for example whether they contain personal data)?

• Did you consider ways to develop the AI system or train the model without or with minimal use of potentially sensitive or personal data?

• Did you build in mechanisms for notice and control over personal data depending on the use case (such as valid consent and possibility to revoke, when applicable)?

• Did you take measures to enhance privacy, such as via encryption, anonymization and aggregation?

• Where a Data Privacy Officer (DPO) exists, did you involve this person at an early stage in the process?

Quality and Integrity of Data

• Did you align your system with relevant standards (for example ISO, IEEE) or widely adopted protocols for daily data management and governance?

• Did you establish oversight mechanisms for data collection, storage, processing and use?

• Did you assess the extent to which you are in control of the quality of the external data sources used?

• Did you put in place processes to ensure the quality and integrity of your data? Did you consider other processes? How are you verifying that your data sets have not been compromised or hacked?

Access to Data

• What protocols, processes and procedures did you follow to manage and ensure proper data governance?

  • Did you assess who can access users’ data, and under what circumstances?

  • Did you ensure that these persons are qualified and required to access the data, and that they have the necessary competences to understand the details of data protection policy?

  • Did you ensure an oversight mechanism to log when, where, how, by whom and for what purpose data was accessed?

4. Transparency

Traceability

• Did you establish measures that can ensure traceability? This could entail documenting the following methods:

  • Methods used for designing and developing the algorithmic system:

    • Rule-based AI systems: the method of programming or how the model was built;

    • Learning-based AI systems; the method of training the algorithm, including which input data was gathered and selected, and how this occurred.

  • Methods used to test and validate the algorithmic system:

    • Rule-based AI systems; the scenarios or cases used in order to test and validate;

    • Learning-based model: information about the data used to test and validate.

  • Outcomes of the algorithmic system:

    • The outcomes of or decisions taken by the algorithm, as well as potential other decisions that would result from different cases (for example, for other subgroups of users).

Explainability

• Did you assess:

  • to what extent the decisions and hence the outcome made by the AI system can be understood?

  • to what degree the system’s decision influences the organization’s decision-making processes?

  • why this particular system was deployed in this specific area?

  • what the system’s business model is (for example, how does it create value for the organization)?

• Did outcome that all users can understand? you ensure an explanation as to why the system took a certain choice resulting in a certain

• Did you design the AI system with interpretability in mind from the start?

  • Did you research and try to use the simplest and most interpretable model possible for the application in question?

  • Did you assess whether you can analyze your training and testing data? Can you change and update this over time?

  • Did you assess whether you can examine interpretability after the model’s training and development, or whether you have access to the internal workflow of the model?

Communication

• Did you communicate to (end-)users – through a disclaimer or any other means – that they are interacting with an AI system and not with another human? Did you label your AI system as such?

• Did you establish mechanisms to inform (end-)users on the reasons and criteria behind the AI system’s outcomes?

  • Did you communicate this clearly and intelligibly to the intended audience?

  • Did you establish processes that consider users’ feedback and use this to adapt the system?

  • Did you communicate around potential or perceived risks, such as bias?

  • Depending on the use case, did you consider communication and transparency towards other audiences, third parties or the general public?

• Did you clarify the purpose of the AI system and who or what may benefit from the product/service?

  • Did you specify usage scenarios for the product and clearly communicate these to ensure that it is understandable and appropriate for the intended audience?

  • Depending on the use case, did you think about human psychology and potential limitations, such as risk of confusion, confirmation bias or cognitive fatigue?

• Did you clearly communicate characteristics, limitations and potential shortcomings of the AI system?

  • In case of the system's development: to whoever is deploying it into a product or service?

  • In case of the system's deployment: to the (end-)user or consumer?

Subscribe now

Share

5. Diversity, Non-Discrimination and Fairness

Unfair Bias Avoidance

• Did you establish a strategy or a set of procedures to avoid creating or reinforcing unfair bias in the AI system, both regarding the use of input data as well as for the algorithm design?

  • Did you assess and acknowledge the possible limitations stemming from the composition of the used data sets?

  • Did you consider diversity and representativeness of users in the data? Did you test for specific populations or problematic use cases?

  • Did you research and use available technical tools to improve your understanding of the data, model and performance?

  • Did you put in place processes to test and monitor for potential biases during the development, deployment and use phase of the system?

• Depending on the use case, did you ensure a mechanism that allows others to flag issues related to bias, discrimination or poor performance of the AI system?

  • Did you establish clear steps and ways of communicating on how and to whom such issues can be raised?

  • Did you consider others, potentially indirectly affected by the AI system, in addition to the (end)-users?

• Did you assess whether there is any possible decision variability that can occur under the same conditions?

  • If so, did you consider what the possible causes of this could be?

  • In case of variability, did you establish a measurement or assessment mechanism of the potential impact of such variability on fundamental rights?

• Did you ensure an adequate working definition of “fairness” that you apply in designing AI systems?

  • Is your definition commonly used? Did you consider other definitions before choosing this one?

  • Did you ensure a quantitative analysis or metrics to measure and test the applied definition of fairness?

  • Did you establish mechanisms to ensure fairness in your AI systems? Did you consider other potential mechanisms?

Accessibility and Universal Design

• Did you ensure that the AI system accommodates a wide range of individual preferences and abilities?

  • Did you assess whether the AI system usable by those with special needs or disabilities or those at risk of exclusion? How was this designed into the system and how is it verified?

  • Did you ensure that information about the AI system is accessible also to users of assistive technologies?

  • Did you involve or consult this community during the development phase of the AI system?

• Did you take the impact of your AI system on the potential user audience into account?

  • Did you assess whether the team involved in building the AI system is representative of your target user audience? Is it representative of the wider population, considering also other groups who might tangentially be impacted?

  • Did you assess whether there could be persons or groups who might be disproportionately affected by negative implications?

  • Did you get feedback from other teams or groups that represent different backgrounds and experiences?

Stakeholder Participation

• Did you consider a mechanism to include the participation of different stakeholders in the AI system’s development and use?

• Did you pave the way for the introduction of the AI system in your organization by informing and involving impacted workers and their representatives in advance?

6. Societal and Environmental Well-Being

Sustainable and Environmentally Friendly AI

• Did you establish mechanisms to measure the environmental impact of the AI system’s development, deployment and use (for example the type of energy used by the data centres)?

• Did you ensure measures to reduce the environmental impact of your AI system’s life cycle?

Social Impact

• In case the AI system interacts directly with humans:

  • Did you assess whether the AI system encourages humans to develop attachment and empathy towards the system?

  • Did you ensure that the AI system clearly signals that its social interaction is simulated and that it has no capacities of “understanding” and “feeling”?

• Did you ensure that the social impacts of the AI system are well understood? For example, did you assess whether there is a risk of job loss or de-skilling of the workforce? What steps have been taken to counteract such risks?

Society and Democracy

• Did you assess the broader societal impact of the AI system’s use beyond the individual (end-)user, such as potentially indirectly affected stakeholders?

7. Accountability

Auditability

• Did you establish mechanisms that facilitate the system’s auditability, such as ensuring traceability and logging of the AI system’s processes and outcomes?

• Did you ensure, in applications affecting fundamental rights (including safety-critical applications) that the AI system can be audited independently?

Minimising and Reporting Negative Impact

• Did you carry out a risk or impact assessment of the AI system, which takes into account different stakeholders that are (in)directly affected?

• Did you provide training and education to help developing accountability practices?

  • Which workers or branches of the team are involved? Does it go beyond the development phase?

  • Do these trainings also teach the potential legal framework applicable to the AI system?

  • Did you consider establishing an ‘ethical AI review board’ or a similar mechanism to discuss overall accountability and ethics practices, including potentially unclear grey areas?

• Did you foresee any kind of external guidance or put in place auditing processes to oversee ethics and accountability, in addition to internal initiatives?

• Did you establish processes for third parties (e.g. suppliers, consumers, distributors/vendors) or workers to report potential vulnerabilities, risks or biases in the AI system?

Documenting Trade-offs

• Did you establish a mechanism to identify relevant interests and values implicated by the AI system and potential trade-offs between them?

• How do you decide on such trade-offs? Did you ensure that the trade-off decision was documented?

Ability to Redress

• Did you establish an adequate set of mechanisms that allows for redress in case of the occurrence of any harm or adverse impact?

• Did you put mechanisms in place both to provide information to (end-)users/third parties about opportunities for redress?

Conclusion

Thank you for reading my publication, and if you consider it helpful or inspiring, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

This is a four part series. This one is the third part of the series. You can find the others here:

• Trustworthy Artificial Intelligence (Part 1)

• Trustworthy Artificial Intelligence (Part 2)

• Trustworthy Artificial Intelligence (Part 3)

• Trustworthy Artificial Intelligence (Part 4)

In my previous articles I’ve begun exploring the approach taken by the European Union towards the definitions and regulations for the development of a Trustworthy Artificial Intelligence1.

If we are to delegate with confidence important tasks to Artificial Intelligence systems, we need to make sure those are trustworthy, from their design, through their implementation to their operation and maintenance.

But, as we have already analyzed, it is necessary to globally agree on what it means to be “trustworthy”, because Artificial Intelligence systems will be cross-border applications.

If every jurisdiction has different regulations and requirements, it will make very difficult for AI developers to offer their services in an homogeneous way, potentially creating markets where this kind of technology might not be available at all, or just be available in a limited fashion, creating undesired inequalities and disadvantages.

The European Union has started this discussion from the abstract Fundamental Rights present in its charter, moving down to principles, then requirements for its real-world realisation and, finally, ways to assess the technology while being developed and once in operation.

In this article we will discuss the requirements that need to be satisfied by any Artificial Intelligence so it can be deemed trustworthy, and the methods of realisation to achieve that (both technical and non-technical).

Requirements of Trustworthy AI

Trustworthiness in Artificial Intelligence must cover the whole lifecycle of its existence: design, development and operation. It is not enough to perform quality control of the final product, because so many things can go wrong during the design and development, and won’t be discernible until it is too late (when the damage has already been done).

According to the European Union expert group, four types of stakeholders will participate in this process, to help guarantee an AI is trustworthy:

• Designers/Developers: This group is responsible of designing and building the AI tools, and should make sure to follow the guidelines and regulations already in place.

• Deployers: This group is responsible of selecting and deploying AI tools for use in their organizations or by the general public. They should make sure those tools have been designed and built according to the guidelines and regulations.

• Users: This group makes direct use of the AI tools, so they are responsible of demanding these tools are compliant.

• Broader society: This group is not a direct user of the AI tools, but as with the Users, they are responsible of exerting pressure in the Deployer and Designers/Developers group, so those tools are compliant.

According to the EU Expert Group on Ethical and Trustworthy AI, the following are the minimum requirements for guaranteeing an AI software can be considered Trustworthy:

• Human Agency and Oversight

• Technical Robustness and Safety

• Privacy and Data Governance

• Transparency

• Diversity, Non-Discrimination and Fairness

• Societal and Environmental Wellbeing

• Accountability

Each of those are of equal importance, but as we discussed in my previous article, it is possible for them to be at odds in certain situations. If that happens, careful decision making will be necessary, to ensure the result is the best possible under those circumstances.

We will review each one of them.

1. Human Agency and Oversight

We have seen that “Respect for Human Autonomy” is one of the principles that need to be protected during the development of AI systems. This means that AI should not be allowed to take decisions or actions on its own, ignoring the required human agency and oversight.

Fundamental Rights

We are entering the Era of AI. The way society works, our jobs, our personal relationships are going to be extensively affected, just as it happened during the previous Industrial Revolutions.

We cannot predict all the possible scenarios where the usage of AI can be perjudicial to human beings, affecting our Fundamental Rights, so it is necessary to stay aware and vigilant, to ensure any issue that arises is dealt with promptly.

Governments, Private Sector, Academia and the General Public must keep their eyes open, especially during this initial period, where we have lots of questions and so little answers.

Human Agency

Human Agency means we are the users of AI, and not the other way. AI systems must support human efforts, and we humans need to have the final say in the decision of using or not whatever the AI produced.

We have seen situations where software systems have been used to manipulate population (like the Cambridge Analytica scandal). AI systems can make this kind of undesirable behavior easier to achieve, so active vigilance needs to be established to prevent it.

Human Oversight

Human Oversight means that we, as humans, are always “in the loop”. It must not be allowed the creation of systems where the AI decisions cannot be monitored, evaluated and if necessary, overridden by human operators.

An example of this is the prevention of the development and deployment of Lethal Autonomous Weapon Systems (LAWS). Think of an automated machine gun turret, AI-powered, that can decide when to kill and when not to. Or the deployment of “killer robot dogs2” (like those cute quadruped mechanical “dogs” we have already seen, but armed with lethal weapons and with the ability to act by themselves, without human oversight).

2. Technical Robustness and Safety

Trustworthy AI requires that any AI system developed upholds the “Principle of Prevention of Harm”. Technical Robustness means that the AI systems are designed and developed in a way that no intended nor unintended harm can come from its utilization.

Furthermore, once deployed, AI systems must be assessed constantly to ensure they still operate under the assumptions they were designed for, and if those environmental assumptions have changed, they must be adjusted accordingly.

Resilience to Attack and security

Artificial Intelligence systems will be subject to malicious attacks. They have already been in the past (just remember what happened to Microsoft’s Tay experiment, who became a racist and offensive chatbot in less than a day of being online3, because of its interactions with the people.

Adversarial Painting4 is a technique that allows attackers to create patterns that make impossible for Computer Vision systems to identify correctly elements they have been trained for. Adversarial Patterns applied to a road marker, for example, can cause accidents on AI-powered vehicles, because it won’t be able to recognize them.

A Risk Management approach needs to be taken, to protect the data and operation of the AI systems, avoiding tampering. Also constant monitoring of the systems health and assumptions must be done, to perform proactive adjustments before harm can be done.

Fallback Plan and General Safety

AI systems will be utilized in situations where harm can come in case of malfunction. Fallback systems must be in place, that allow a human to take over, or a more traditional (and proven) system can provide a validating input.

We are at the dawn of AI. While we gain real world experience of the application of AI to everyday processes, we must ensure the proper risk management of those situations: the greater the risk, the greater the security measures included by design.

Accuracy

It is well known that the current iteration of Artificial Intelligence systems, based on Large Language Models (LLMs) can be prone to have “hallucinations”: constructing an answer that can be statistically probable, but is completely wrong.

Future Artificial Intelligence paradigms, perhaps closer to the promised Artificial General Intelligence (AGI), will be able to be more precise, to properly work around conjectures and have a general world model that will allow it to have the “common sense” that it sorely lacks today.

In the meantime, we need to have in place methods and processes to validate the correctness of the answers received, and reduce the risk of acting on inaccurate information.

Reliability and Reproducibility

Results generated by Artificial Intelligence systems should be reliable and reproducible. Reliability means AI systems are capable of producing results under every circumstance required of it, while Reproducibility means the AI system is capable of producing the same behavior under the same conditions.

Subscribe now

Share

3. Privacy and Data Governance

Privacy is one of the Fundamental Rights recognized by the European Union charter, and the advent of AI systems pose an undeniable risk to this right, and the “Principle of Prevention of Harm”. The same can happen as a product of improper Data Governance.

Privacy and Data Protection

Artificial Intelligence systems are capable of sifting through enormous amounts of data, both during their training process and operation.

This data can come from many sources, and some of them might include personal or sensitive information, that after the model creation will become accessible to anyone who requests it.

Copyrighted works, either texts, images, music, etc. can also be processed by Artificial Intelligence systems as part of their training, enabling them to create similar works that, without being a derivative themselves, could be considered a copyright infringement.

This is currently a matter of intense debate, and several instances of undesired events have already occurred5 because of poor personal and identifying data management, and copyright infringement.

Quality and Integrity of Data

Data collected for AI training can have biases, and those biases will become part of the system’s answers and results. Special care must be taken to avoid that from happening, as it can be difficult to fix once the model is in operation.

Even worse situations can arise with AI systems capable of automated learning, as those can acquire improper information, perhaps maliciously fed into them. As mentioned before, we already have examples of this (the Microsoft’s Tay scenario).

Access to Data

If an Artificial Intelligence system is going to be fed with confidential and sensitive information (like personal data), it must be designed, developed and deployed in a way that guarantees proper data access protocols are followed.

4. Transparency

The “Principle of Explicability” makes necessary that any AI system is transparent about its data, its algorithms, design, construction, and any business model it might follow.

Traceability

Given the current state of AI, and the way LLMs work, it is still very necessary to validate the answers they provide, or we risk acting on “hallucinations”.

Traceability of the way an answer was achieved, through the algorithms and all the way to the data sources, is required for a Trustworthy AI.

This way it will be possible to pinpoint errors, fix the causes, and avoid future occurrences of the same.

Explainability

Humans and AI systems will work together to produce results for the organizations employing them. As such, the concept of Explainability must cover both parts: AI and Human.

Technical Explainability in some cases is at odds with the capabilities or performance of the AI system (as it introduces complexities in its design that can make it slower, less powerful and harder to maintain). But it is necessary if we are going to be able to trust its results.

In the other hand, it is also necessary to provide transparency on which parts of an organization’s decision processes are affected by AI systems, and why it has been deemed conveniente to use them.

Communication

As humans, we have the right to know when we are interacting with an AI system or another human. Current AI systems might not be able to fully impersonate a human being, but that might change soon.

And even while knowingly interacting with an AI system, we have the right to know what are its limitations and capabilities, so we can better choose how and when to use the answers and knowledge provided, minimizing risks.

5. Diversity, Non-Discrimination and Fairness

Inclusion and Diversity regarding AI systems shouldn’t be an afterthought, but be present in the whole lifecycle of its creation and operation. This will help uphold the “Principle of Fairness”.

Avoidance of Unfair Bias

AI systems can be vulnerable to have unfair biases caused by biased data ingestion or biased algorithms (it has already happened, as can be seen in this MIT Technology Review article6, which narrates how an AI-powered Police Station had to deactivate its system because of racial bias).

The best way to prevent this from happening is to have processes in place that remove unwanted biases from the data, and also diverse hiring processes, as it will make harder for these biases to go inadvertently into the final product.

Accessibility and Universal Design

AI systems will be used by many diverse population groups, so Accessibility must be kept in mind during its whole lifecycle. Universal Design as a concept and guidance is useful to achieve this goal.

Stakeholder Participation

Everyone involved in the AI system lifecycle should participate in some way in its design, to make sure it is built according their needs and doesn’t pose a threat or risk.

In the same way, once it is in operation, feedback from those stakeholders will be key to keep improving it over time, adding functionality or fixing things that doesn’t work.

Subscribe now

Share

6. Societal and Environmental Wellbeing

AI systems can have both a positive and negative impact in society, the environment and the general wellbeing of mankind, both present and future. It is important to ensure the “Principles of Fairness and Prevention of Harm” are considered during its lifecycle.

Sustainable and Environmentally Friendly AI

AI systems will enable us to perform analysis beyond our current capabilities, and this will help us to tackle some of the hardest problems we are facing today, like our response to the Climate Change.

But, at the same time, AI systems consume enormous amounts of processing power both for their training and operation, and that can also become a problem on itself.

Social Impact

AI systems are already permeating many aspects of our society: students and teachers using them to create content, lawyers using them to summarize documents, businesses using them to serve online customers, etc.

And this is only the beginning. We must try to predict the potential impact, to avoid it destroys vital elements in our society, like social skills, relationships, agency, etc.

Society and Democracy

AI systems can also be negatively used to manipulate populations, especially during electoral processes. It is important to monitor their usage to avoid this from happening.

7. Accountability

The “Principle of Fairness” brings as a necessity the requirement of Accountability. It is necessary to provide accountability for every part of the lifecycle of the AI systems we develop, if we are to have Trustworthy AI.

Auditability

Auditability is key for a Trustworthy AI, as it allows the assessment of an AI system’s data, processes, algorithms, etc. It not necessary to openly expose sensitive information like business models, data models, weights, etc. Those can be assessed using proper access methodologies, and for the routine processes, Observability techniques can be used.

Minimization and Reporting of Negative Impacts

AI systems and their controlling organizations must not be able to prevent users, employees, stakeholders or anyone who detects a misbehavior in the AI to report it to the proper channels in place.

The focus must be on minimizing the potential damage caused by an AI malfunction, by enabling a fast and streamlined way to report it, so proper actions can be taken immediately.

Trade-offs

As mentioned before, there will be situations where tensions will arise between those requirements. If a principle or requirement is given precedence over another, it must be done in a reasoned and well documented way, because at some point it might be necessary to support that decision.

Those trade-offs must be analyzed from the ethical and risk management perspectives, and if it results impossible to solve the situation in an ethically acceptable way, the development or deployment must be aborted.

Redress

Important for the trustworthiness of an AI system is the knowledge in advance that adequate mechanisms for redress exist, in case a negative impact occurs.

Technical and Non-Technical Methods to Realise Trustworthy AI

Artificial Intelligence systems might not be new as a technology, but their introduction to many business domains is definitely a recent occurrence.

The way in which AI systems are developed is also constantly evolving, to adapt to new challenges found along the way, or make use of new advantages and capabilities of hardware, algorithms, etc.

And the AI systems themselves have also been evolving, at a breakneck speed, posing a challenge to organizations trying to decide which option available to adopt for their business purposes.

Both models are following a cyclical-iterative process, which require constant assessments to keep improving over time and not being left behind. But at the same time, this constant assessments should try to increase AI trustworthiness.

Trustworthy AI can be realised by both Technical and Non-Technical methods and we will briefly mention some of them, that are in no way exhaustive.

Technical Methods

The Technical Method to ensure a Trustworthy AI must be ingrained in the whole AI system lifecycle: design, development, deployment and operation.

Architectures for Trustworthy AI

Trustworthy AI systems will have the requirements directly integrated into it from its design and architecture, thus making impossible for it to enter undesired states or to provide inappropriate answers.

AI systems capable of continuous and unsupervised learning are at risk of acquiring by themselves data and training that will produce undesired results. To avoid this, they must be able to identify those rogue elements and states during their “Sense-Plan-Act” cycle, and filter them out accordingly.

Ethics and Rule of Law by Design (X-by-Design)

In the same way that several “by-Design” methods have been adopted (like “Privacy-by-Design”, or “Security-by-Design”), it is necessary to adopt a Trustworthy-by-Design and Ethics-and-Rule-of-Law-by-Design.

This approach will ensure compliance at every level of the AI system lifecycle.

Explanation Methods

If we are going to trust an AI system’s result, we need to be able to explain how it arrived to that answer. This need has fostered the development of a new field of research: Explainable AI (XAI).

XAI employs diverse programming techniques during the design and construction of AI systems that allows to trace the flow of “reasoning” that leads to a specific answer (for example, identifying which “neurons” fire up as a result of the prompt requested).

This is very important for the current iteration of AI systems, based on LLMs trained using huge amounts of data, because a small variation in a neural network weight can produce a highly different type of answers, and it is necessary to properly map this process.

Testing and Validating

AI systems based on the current trend (LLMs) are non-deterministic. This means that for the same input they can produce different answers (how different it will be is a matter of certain parameters, like “temperature” (randomness level).

This means that common deterministic testing techniques for software are not enough. Tests and validations must not only happen during the design and construction phases, but also during the whole operation phase, because changes in the context where the AI is used can lead to unexpected results, that might need to be dealt with.

Quality of Service Indicators

Software systems commonly are developed while trying to achieve certain Quality of Service (QoS) Indicators, that measure performance, security, reliability, etc.

AI systems should have all of these, and also QoS indicators around the accuracy level of the answers they provide, and all the trustworthiness parameters we have mentioned so far and more.

Subscribe now

Share

Non-Technical Methods

Non-Technical methods for realising and maintaining Trustworthy AI systems are also necessary during their whole lifecycles. The EU Expert Group on AI listed the following:

Regulation

Regulations that will allow AI to be trustworthy are starting to appear all around the world. But they will need to be revised constantly, as the AI landscape evolves (and it is evolving rapidly).

Codes of Conduct

Not only governments can be held responsible of building and deploying Trustworthy AI: corporations can bring trustworthiness concepts into their internal Codes of Conduct too.

Standardisation

Many standards exist in different industries, and those help to manage expectations about quality, performance, features, etc.

Eventually the AI landscape will adopt Trustworthy AI standards that will be adopted by the different stakeholders, making it easier for everyone to create and use the AI systems without tinkering with individual regional regulations.

Certification

Along with Standardization, we can expect Certification to become one of the tools to achieve widespread adoption of Trustworthy AI practices, during the whole AI systems lifecycle.

Accountability via Governance Frameworks

Certification and Standardization help prove to external stakeholders that the AI systems developed or deployed are compliant with Trustworthy AI best practices.

But the organization should have an internal governance frameworks, with executives at the proper level responsible of the ongoing compliance, and board members related to Ethics in AI, if necessary.

Also keeping in touch with oversight groups and public stakeholders that might provide guidance or external experience can be useful.

Education and Awareness to Foster an Ethical Mindset

Trustworthy AI is responsibility of all the stakeholders: designers, developers, deployers, users, etc. This means everyone needs to be literate about the importance of AI Ethics and Trustworthiness, so they can fulfill their role in keeping AI systems compliant.

Stakeholder Participation and Social Dialogue

Learning from everyone involved about their perspectives on the use of AI will be a key factor to create AI systems that benefit the most the whole society.

This requires a widespread stakeholder participation and dialogue, which must be either led by governments or interested parties, like the organizations developing or deploying AI tools.

Diversity and Inclusive Design Teams

As we mentioned before, it is very important to have diverse and inclusive design and implementation teams. Having gender, culture, age, ability, professional, etc. diversity can help to detect, avoid and correct biases that could appear during the AI system lifecycle, helping to increase its trustworthiness.

What comes next

Stay tuned, as I’ll be posting the fourth part “Assessing Trustworthy AI” on my next article!

Thank you for reading my publication, and if you consider it is being of help to you, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

(Source: Bing Image Creator)

This is a four part series. This one is the second part of the series. You can find the others here:

• Trustworthy Artificial Intelligence (Part 1)

• Trustworthy Artificial Intelligence (Part 2)

• Trustworthy Artificial Intelligence (Part 3)

• Trustworthy Artificial Intelligence (Part 4)

Continuing our discussion about Trustworthy Artificial Intelligence (check the first post here1), now we must “start with why” (like Simon Sinek probably would say). Why is it very important for us to make sure any Artificial Intelligences we develop are Trustworthy, and what means to be Trustworthy?

We, humans, are the ones attempting to create an Artificial Intelligence capable of rivaling ourselves in cognitive abilities (and perhaps surpassing us). Our motivations, beyond the common human impulse of doing it just because we want to prove we can (think of climbing the Mt. Everest) are also of self-benefit.

We want to make our lives easier, and we have been achieving that for quite some time. From Animal Power (when we domesticated oxen, horses and etc. to supplement “manpower”), through the Industrial Revolution (steam power), Electricity and Mass Production, Electronics and Digitalization, all the way to the current Fourth Industrial Revolution: Artificial Intelligence and Robotics.

So, the development of AI is expected to be “Human Centric”. We are the beneficiaries of its creation. But AI is different from all our previous endeavors. Once we get it done, it won’t be just a tool, but it will have conscience, self awareness and most probably its own agency. It might set out to find its own purpose in existence.

AI is at its infancy. Remember the Narrow AI definition. It still can be considered a tool. But it is a tool that is increasingly becoming capable of taking decisions and actions by itself that can potentially affect our lives. So its development must include measures to assure it will exist to help us, by design, instead of just trying to fix it as an afterthought (once something bad has happened).

This brings us to the concept of “trustworthiness” applied to Artificial Intelligence. We need to create a Trustworthy AI. By design, not by chance. And this means having a good definition and framework for that, because the AI efforts are happening all around the world at the same time. So we all must agree on what means to have a Trustworthy AI, how to achieve it with our current tools and techniques, and how to assess if an AI is trustworthy before committing our wellbeing and safety to it.

On this post we will explore how the European Union High Level Expert Group defined the Ethical and Trustworthy AI, which in turn served as the basis for the current European Union Artificial Intelligence Act on the verge of being voted and approved. On the next ones we will review how it is proposed to achieve a Trustworthy AI, and then how to assess and evaluate an AI to determine if it can be considered Trustworthy according to these guidelines.

Trustworthy AI

According to the Ethics guidelines for trustworthy AI2, Artificial Intelligence should be Human Centric. We have already discussed this. We want to maximize our potential benefits while minimizing the associated risks.

This requires not only assessing the final product (the AI tools and systems produced) but the whole development process, in the same way we not only randomly evaluate samples of food already produced, but we also seek to certify the process through it is produced, so it reliably continues to do so in an acceptable way.

So, when we create a Trustworthy AI, and once it exists and operates, it must comply with three components:

• It should be Lawful

• It should be Ethical

• It should be Robust

Each of those is necessary, but not sufficient to ensure trustworthiness. All of them must exist at the same time. Which can be difficult, as it will happen that in some situations there will be conflicts between them: Robustness might be made more difficult because of Lawfulness and Ethical requirements, etc.

When the EU Ethical Guidelines were developed, back in 2019, the AI explosion that we have today wasn’t even thought as a possibility. So the idea was to make them “voluntary” for anyone working on or with AI. Things have changed now. AI has become widespread, and the associated risks have skyrocketed, making it imperative to provide a regulatory framework to deal with it.

This sparked the need to create the European Union Artificial Intelligence Act (explained here3), and fast track it into approval, because AI research, applications and usage won’t wait, and we have already witnessed some examples of misuse, unlawful use cases or ethically challenged situations. The fact that AI development is a cross borders business doesn’t help.

Let’s review each of the AI Trustworthiness components.

Lawful AI

Artificial Intelligence systems are created by people or organizations in a country. Each country has (or doesn’t have) regulations about AI. Then such AI systems are consumed either in the same country or another. Which has its own AI regulations. Those regulations probably will cover not only the AI system’s behavior and capabilities, but also how it was developed and how the service is provided (remember our previous example about not only certifying the final food product, but also the production process for quality and safety).

Subscribe now

Share

Artificial Intelligence systems will have to operate in compliance to multiple legal frameworks, and it is the responsibility of those building them to check the requirements of the many markets they will target to make sure they remain lawful.

The AI boom is has stirred up the legal machinery on every country and economic bloc, because of both real and perceived potential benefits and risks. This means that anyone involved in AI needs to keep their eyes open to potential changes in regulations, as those will be constantly updated based on experience and new developments.

In the end, a Trustworthy AI needs to keep legal compliance, because no one will trust it enough to use it if by doing so you’ll get in legal problems. This means constant reviews and oversight of such compliance, which can become cumbersome and costly, but necessary.

Ethical AI

As History painfully shows us, Lawful doesn’t always means Ethical (think of slavery). Technology moves much faster that laws. Laws and regulations are constantly playing catch up with new developments, and in the meantime, undesirable things happen. We can find tons of examples in our current world.

So it becomes of utmost importance the use of proper Ethics in every aspect of the AI development and utilization. Laws may be slow to come, but Ethical principles will be already there and should guide every decision we take on this field.

When thinking about how to apply Ethics to AI, we must consider the following situations, amongst others:

• The purpose itself of why an AI system is being developed (like the design and construction of AI powered Autonomous Weapons Systems)

• The development process. It is necessary to make sure the data employed in the AI training is not biased, or the AI behavior will be. This has happened already, like shown on this Technology Review article4.

• Also under the development process, the algorithms themselves can be biased, not only the data. Software Developers are human beings. They have biases, that might seep into the code and end up polluting an AI behavior.

• The usage process itself. An AI system might be conceived and developed ethically, but then used in an unethical way. For example, using facial recognition AI to create Social Score systems, like the one existing today in China.

Robust AI

When we talk about a Robust AI, we mean it’s been designed and built in a way that ensures no unintentional harm can come from it. This is mostly technical, but it is closely intertwined with the ethical aspect of AI.

The AI purpose, its development process and its usage must be conducted safely, securely and reliably

Foundations of Trustworthy AI

When we talk about having a Trustworthy AI that is Human Centric, upholds the law, it is ethical in nature and of robust design, construction and operation, we need a starting point.

This starting point should be based on Human Rights and concepts that are transversal to all the countries where the AI regulations being developed are expected to be implemented.

Those Fundamental Rights will guide the development of Ethical Guidelines for every aspect of Artificial Intelligence implementations, and in particular we will explore the ones proposed by the European Union through its committees and expert groups.

Subscribe now

Share

Fundamental Rights

Let’s review the Fundamental Rights the EU Expert Group on AI has stated as necessary for the development of Trustworthy AI.

Respect for Human Dignity

We humans have an “intrinsic worth”. We are subjects, not objects. The introduction of mainstream AI increases the risk of having governments and corporations treating humans as goods or merchandise to be identified, classified, and assigned a value according to their interests. Special care must be taken to avoid this from happening (as it already happens on countries like China).

Freedom of the Individual

The use of AI should increase the human freedom, not diminish it. Common expectations are that it should empower humans to get more done with less effort, increasing our discretionary time and our productivity. But freedom of choice must be protected: people who doesn’t want to use shouldn’t be forced into it, nor be negatively impacted by that decision.

The other way around is also important: AI should be available to everyone, if they choose to use it. If not, we would be introducing a third type of illiteracy: the AI Illiteracy (so far we already have the normal one, the digital illiteracy and now this).

Respect for Democracy, Justice and the Rule of the Law

The ability of AI to handle enormous amounts of information can lead to it being used to manipulate the population into political and social decisions that might not be in their best interests.

We have already seen examples of this, even without the current Generative AI trend (think of the Cambridge Analytica case). Unlawful and unethical use of AI is something that needs to be carefully monitored.

Equality, Non-Discrimination and Solidarity

Current AI is vulnerable to acquire biases, either by its programming or its training. This is something that needs to be prevented by design and monitored after it enters mainstream use.

AI shouldn’t facilitate discrimination nor exclusion. Think of using AI to sift through millions of personal profiles to determine who should be hired and who shouldn’t. Or through medical records, to determine who should get insurance and who shouldn’t.

Citizens’ Rights

Citizens have a number of rights, according to their own nations. AI should help them to have better access to those rights, instead of increasing friction. Governments can make use of AI to streamline public services and become more efficient.

But also it should be prevented the use of AI to allow governments and totalitarian regimes to exert unethical surveillance and control over its citizens.

Ethical Principles

AI design, development and operation should be guided by strong ethical principles that ensure it is used to benefit humans in general (Human Centric approach).

Those ethical principles can come from many different sources, like fundamental rights charters, constitutions, etc.

In particular, the following ethical principles are based on the European Union’s Treaties and Charter. From those, the European Group on Ethics in Science and New Technologies proposed a set of nine basic principles. And the High Level Expert Group responsible for the AI Ethics derived four from those, that are discussed below.

Subscribe now

Share

Respect of Human Autonomy

AI should be developed with a Human Centric approach (it is a tool to help and support us, and not the other way around).

We, as humans, shouldn’t be under AI control and oversight. Our decisions must come first, and we shouldn’t be subject to AI decisions we cannot overrule.

Prevention of Harm

It’s been some time since Isaac Asimov’s “Three Laws of Robotics” came to be. Those should have been labeled instead “Three Laws of AI”, because the robotic behavior will be mandated by its governing AI.

As it’s been repeatedly stated, AI should be deployed to benefit humans, in a Human Centric way.

This means, above all, that AI should not cause harm, or exacerbate it. Human dignity and integrity must be ensured, and by no means it should be used by governments or organizations, in detriment of individuals (like the insurance and Human Resources examples previously provided).

AI technical robustness must also ensure no intended or unintended harm can come from the use of AI.

Fairness

AI should be fair. This means that its design, development/training and operation must not be subject to biases that might place people in unfair situations, purposefully or inadvertently.

This is particularly relevant when considering people with disabilities, minorities, or any kind of disadvantage. AI should provide equal opportunities to all, and its costs and benefits should be distributed with fairness amongst everyone.

And if, for any reason, people are harmed by either AI or someone operating it, they must be able to pursue proper remediation. Accountability in the use of AI will be key to establish trust on it.

Explainability

For some time, Neural Network based AI have been deemed a “black box”, where the answers they provide cannot be fully explained nor traced.

As it was explained in my previous post, this is still the case, because of the complexity of deep neural networks. But intense work is being done to improve the Generative AI Explainability, as this is a key requirement to create a Trustworthy AI.

As mentioned in the Fairness section, accountability is key for the development of trust. And this requires proper traceability of the process followed to arrive to answers and decisions provided by AI.

What to do when there are conflicts between principles

Under some circumstances, it will be unavoidable to find conflict between the stated principles.

Just from Asimov’s “Three Laws of Robotics” we can find one: Human Autonomy can come in conflict with Prevention of Harm, if a Tesla vehicle chooses to override the driver input on the accelerator pedal to avoid a fatal accident where the driver would have run over a sprinting child that he didn’t see, but the car did.

Or most notably, Large Language Model explainability can go against both the model performance and capabilities (not to mention operational costs) as it has to add processing and data complexity to achieve such explainability (and sometimes not achieving it completely). The lack of Explainability can go directly against all of the other principles, like Fairness (inability to trace the source of a bias), Prevention of Harm (cannot reliably ensure it won’t perform harmful actions) or Respect of Human Autonomy.

What comes next

Stay tuned, as I’ll be posting the third part “Realisation of Trustworthy AI” on my next article!

Thank you for reading my publication, and if you consider it is being of help to you, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

Subscribe now

Share

References

This is a four part series. This one is the first part of the series. You can find the others here:

• Trustworthy Artificial Intelligence (Part 1)

• Trustworthy Artificial Intelligence (Part 2)

• Trustworthy Artificial Intelligence (Part 3)

• Trustworthy Artificial Intelligence (Part 4)

Would you put your life in the hands of a ChatGPT-like Artificial Intelligence decision? A medical decision? A legal decision? Can AI, as we have it today, be trustworthy? What means to be trustworthy?

You would be surprised to know that we are already placing life and death decisions on software systems, all the time. But those are carefully selected situations, where those systems were exhaustively engineered to provide better results than the humans they were replacing.

Think of the control systems in our latest aircraft, where several computers and humans “vote” in a fraction of a second about a maneuver being attempted. The human input determines the intent, but the machines decide how to do it… or if it gets done at all. The aircraft sensors, plugged to the computers, perceive the surroundings faster than humans, and can take a life and death decision to avoid deadly mistakes.

But fatalities caused by aircraft computers misjudging certain situations have already happened (like the Lion Air Flight 610 and others1). This, of course, sparked public outcry and the grounding of the airlines and aircraft models affected. An immediate investigation was launched, to find the root causes and fix them. It became necessary to explain how the computers arrived to the fatal decisions, if those planes were to fly again.

At the same time, we have had a lot of those software systems working reliably for decades, controlling tirelessly, day and night: nuclear reactors, medical ICU hardware, and more recently, some cars, amongst many other things. They just do it better than us.

So, why is Artificial Intelligence any different? This onion, like ogres, has many layers. Let’s peel them one by one.

Current Artificial Intelligence is not THE Artificial Intelligence

Contrary to what most people think, the current type of “artificial intelligence” we have right now doesn’t lead by itself into the common concept of an Artificial Intelligence that sci-fi has planted in everyones’ minds: A computer system capable of “thinking like a human”, one we can have intelligent conversations with, and that can be considered intellectually our equal, to the point of even endangering mankind.

I’ve written extensively about this in my previous post “Artificial Intelligence for Customer Experience2” (you can check it there), but I’m going to do a short summary here anyways.

Computer Scientists have established two categories to classify Artificial Intelligence systems:

• Narrow AI (also known as Weak AI)

• General AI (also known as Strong AI)

Narrow AI has been with us for some time already. Computer Vision systems, like the ones present on Tesla vehicles, capable of navigating through crowded streets. Natural Language Processing systems, like all the translation systems that have been broadly using. Medical Image Processing, now capable of detecting medical conditions far beyond what the expert human eye is capable by itself. And many more.

Narrow AI, by definition, is capable of doing things that we humans do everyday, but faster, more efficiently, and sometimes can do things beyond our capabilities. But only that specific thing. It doesn’t “think”. It cannot generalize nor create conjectures (like Noam Chomsky brilliantly stated on his New York Times article “The False Promise of ChatGPT3”).

In the other hand, General AI (that doesn’t exist yet) is expected to be able to show cognitive capabilities similar to humans, in the results it achieves. As E. Dijkstra, one of the fathers of AI, eloquently said: “The question of whether machines can think is about as relevant as the question of whether submarines can swim”. This means that it is very unlikely that the way machines will achieve AI is going to look like the way we achieve our own intelligence (another question we need to answer yet).

So, why all the ruckus about ChatGPT and the downfall of the Human Society because of it? Well, it is mostly caused by badly informed tech evangelists, marketing people without technology background and unscrupulous media. All of them trying to profit from the hype, of course.

ChatGPT, Bard, etc. belong to a category of Machine Learning systems implemented using a technique called Generative Pretrained Transformers (hence the “GPT”). The simplest way to understand how it works is to think it is like a more complex version of the predictive text completion you have had in your cellphone for a couple of decades already. It feeds on a large amount of data, and it tries to predict, using statistics, what would be the most probable answer for the question (or “prompt” nowadays) you gave to it.

So, yeah, you can keep feeding it curated information (like OpenAI, Google and others are doing right now), or open it to the Internet so it can gobble data at will. But no, you won’t wake up one morning to be surprised by it greeting you with a “Good morning, Dave”, and proceeding to dump you into the cold space. It doesn’t lead to that. It will not turn into a General AI just by expanding its model.

It can be said that ChatGPT and the rest of GPTs are just another iteration of Narrow AI. To achieve a real General AI we still need many breakthroughs. Algorithmic, computational, etc. But it doesn’t mean we cannot reap the benefits that Narrow AI like the Generative ones we have right now can provide. And while at it, we can also prepare our society for when General AI arrives, and ideally learn how to craft it so it follows a human-centric approach.

Subscribe now

Share

Critical Software Systems are carefully engineered. Your ChatGPT bot isn’t

So, we are currently using Narrow AI in many places already. Sometimes critical places. And we have had software running life-and-death stuff for us even without the AI tag on it (google for “Real-time Systems” and you’ll find tons of examples). What makes ChatGPT and its siblings different?

Realtime systems and Narrow AI enhanced systems have been carefully and purposely designed to apply those AI techniques in situations where they can outperform humans. And thoroughly tested. Again and again. And we still have had some serious incidents. Some of them deadly (like the aircraft control one mentioned earlier).

ChatGPT created a level of hype and awareness unheard of in the Consumer AI field. Suddenly everyone was toying with AI. With emphasis in the word “toy”. It wasn’t long before some lawyer attempted to create an argument using ChatGPT, students tried to make it write their essays, etc.

But there is a problem. A big one. Again, Narrow AI doesn’t think. Some people might have the impression it does, because of the articulate level of its responses, but as I already said, it is just a glorified predictive dictionary (and yes, even the AI image creators work that way).

Narrow AI doesn’t “understand the world”. So if it is fed with information that says that the pigs fly, it is going to write a very nice essay about how pigs can fly. Perhaps even a sonnet. You can fool it very easily. But the worst thing is it can fool itself even easier.

We call that “hallucinations”. It happens when a GPT-based AI reaches the limit of its knowledge, and would require the human ability to understand context to provide a careful response. But it doesn’t. Somewhere, someone fed it with a document that said pigs fly and stated pigs aerodynamics, speeds at sea level, etc. And when prompted to write an essay about flying animals, the pigs will be included, with all the detailed data, and even citations. Scary, huh?

But it is even worse. It doesn’t need a malicious agent to feed it with false pig flight capabilities information. It can write nonsense all by itself, even when fed with proper data. Why? Because it is a statistical machine. It is not “redacting” an answer. It is just putting together letters… then words… then paragraphs, that are statistically probable to be an answer, based on the tons of info it has been fed. But if for some reason it ends up being nonsense, its inability to understand that up is up, and why (and when up might not be up) leads it to happily spout a ludicrous answer without hesitation.

So now, instead of having a carefully designed and tested AI system that outperforms humans in very controlled scenarios, we have millions of people toying with AI in an uncontrolled way, in some cases irresponsibly for professional purposes. We are endangering ourselves by our lack of understanding of its real capabilities.

Perhaps you are already aware of those lawyers that presented a ChatGPT-generated argument, which included hallucinated citations of court cases that didn’t exist, and led to the banning of AI usage in court by a Texas judge4.

As usual, it is not the technology per se what causes the damage, but its misuse. And as Murphy Law states: “Anything that can go wrong will go wrong”. So it went. And will keep going, unless we, as a society, manage to place proper guardrails on the way AI works and its usage.

Neural Network-based AI is hard to explain (or even impossible)

The hallucinations and human misuse of AI are impossible to solve? Theoretically, no. Lots of research is being made on how to try to explain the answers given by Generative AI, for many reasons.

To begin with, it is important to understand how it reaches an answer, so if the answer is wrong, it is possible to find a way to fix either the neural network itself, the data, the underlying algorithms, etc.

Also it is important to be able to explain the answer for accountability purposes: if we are going to use AI-generated content as a basis for our own, we need to be able to explain how it reached that conclusion if we are to be asked about it later (not to “blame it”, because we are still responsible of whatever we produced with it, of course).

But let’s address first the elephant in the room. Why is it hard (or impossible) to explain a Neural Network based AI (like the GPT ones)?

First we need to explain a bit about how Neural Networks work, so you can get an idea of why (don’t worry, I’ll try to make it as simple as possible).

This is a “neuron” from an Artificial Neural Network (yes, it looks a lot like a biological neuron because it is inspired on its structure and behavior).

And this is a biological neuron. As you can see, there is quite some resemblance. And it is not by chance. Just as it happens with biological neurons, the artificial one receives inputs from the neurons on the left (schematically, of course).

The Artificial Neuron takes each of the inputs (they can be in the hundreds, thousands or more), multiply each one by its corresponding “weight” (marked with a “w” on each input), sums it all, and adds a value called “bias” (no need to go in detail about this).

Then it applies something called the “Activation Function”, that changes the value so it can be sent into the next Artificial Neuron, and then the result is sent towards the right (the output).

Yes. That’s it. This simple concept is what is behind all of this Generative AI “miracle”.

Subscribe now

Share

But, you are probably asking yourself: OK, Alfredo, don’t fool me. Is this thing capable of learning, writing essays, and getting people into trouble?

Well… yes. Let’s explain how it learns.

When you train an Artificial Neuron, you already know the answer you are looking for. Of course you do. You cannot train someone (or something) if you don’t know the answers beforehand, right?

So your Artificial Neuron starts with a specific Activation Function and a fixed bias (for our purposes, those won’t change during the training process). And also starts with a set of weights (the w’s tied to the inputs).

You run a set of inputs from left to right, and after the neuron calculation is done (inputs multiplied by weights, summed up plus the bias and then the activation function applied) and get the output.

Then you compare the output to the “desired output” (the answer you already know). And if it isn’t the answer you expect, then you proceed to change the weights (the w’s) and try again. Again and again until you get the answer you want.

Now you know which set of weights produce the answer required. That neuron is “trained”.

But you don’t really expect this lone artificial neuron to be able to produce that fancy AI-generated image you saw on X.com (formerly Twitter).

You are right. This is more like the real thing. Neurons are not alone. As in our nervous system, there are A LOT of them. An AI Neural Network can have thousands, tens of thousands or hundreds of thousands, etc.

And each neuron is connected to the next “layer” of neurons. So each line in that diagram is a weight (in fancy GPT terms they are called “parameters”, along the biases). The more layers, the “deeper” the neural network. Now you know why some of these techniques are called “Deep Learning” (because of their “deep neural networks”).

Probably you have heard that the latest ChatGPT, that uses the GPT-4 model, has around 1.7 trillions of parameters. Yep. You read right. 1.7 trillions of weights and biases (those lines in the diagram). And each one of those needs to be trained using the process depicted before.

Now you know why it takes so long to train a Neural Network, and why uses so much processor power (expensive, expensive, expensive), so only behemoths like Microsoft (through OpenAI), Google, Facebook, etc. can train really large models (you have also probably heard the term LLM, that stands for Large Language Models).

So, how do we explain that funky answer my AI neural network produced because of my last prompt? To be 100% precise we would have to backtrack from the answer through each of the parameters to the neurons that participated in the construction of such answer, until we had the complete map of the neurons that fired in sequence to produce it.

As you can probably guess, with 1.7 trillions of parameters this can be near impossible to do. And we are assuming no new “learning” is done between the answer and the explanation attempt. Many techniques that try to reduce the large model to a smaller one (a “local model”) and many others exist.

But, without the need of going into complicated details, by now you can probably assess that we are talking about storing huge amounts of data that will support later attempts of explanation (expensive), processing overhead to calculate and store that data for when it is needed (expensive, expensive), and a lot of extra complexity in case we need to retrain our model, or expand it (expensive x 1000).

And all of this is assuming that the explanation model used is reliable enough for the type of neural network we are trying to explain. Which might not be true. There is a balance between feasibility and trustworthiness of the explanation model, if you are using reductions and simplifications.

So, explaining a Neural Network based AI (like a GPT) isn’t trivial, isn’t easy and isn’t cheap. It might not even be possible. That’s why for so long Neural Networks, once trained, have been considered “black boxes”.

A Trustworthy Generative Artificial Intelligence

The question that immediately arises is: How do we define a Generative Artificial Intelligence that is Trustworthy, so we can make sure we are building them that way? An Artificial Intelligence that we can trust our wellbeing and life on, as we have done before with other software systems.

I’ll try to answer those questions by aligning with the European Union’s concept of “Trustworthy AI”. The EU is probably the current leader in the attempts to create regulations to ensure the development of AI following an ethical path, centered on the human being as its beneficiary.

But this topic is just too big to cover it into a single post, so I’ll split it into three parts, following the EU High Level Expert Group on Artificial Intelligence Report5:

• Foundations of Trustworthy AI

• Realisation of Trustworthy AI

• Assessment of Trustworthy AI

Stay tuned, as I’ll be posting the second part “Foundations of Trustworthy AI” on my next article!

Thank you for reading my publication, and if you consider it is being of help to you, please share it with your friends and coworkers. I write weekly about Technology, Business and Customer Experience, which brings me lately to write a lot also about Artificial Intelligence, because it is permeating everything. Don’t hesitate in subscribing for free to this publication, so you can keep informed on this topic and all the related things I publish here.

As usual, any comments and suggestions you may have, please leave them in the comments area. Let’s start a nice discussion!

Cheers.

When you are ready, here is how I can help you:

By subscribing to my publication and reading this article you have already taken the first steps into redesigning your business to succeed during the Age of AI, .

But if you want to go further, here are several options:

• If you want to start right away, you can join me in my 3-day Future-Proof Entrepreneur Challenge, where you will learn how to analyze your Profit Model (how your business makes money today), look around for the Exponential Technologies you will need to make it work in the Age of AI, and finally create a plan to make it happen.

• You can check my FREE guide “7 Profit Models to Succeed in the Age of AI”.

• And also my FREE guide “7 New Technologies that can wreck your Business like Netflix wrecked Blockbuster”.

The Entrepreneuring AI is free today. But if you enjoyed this post, you can let me know that my content is valuable by fueling my coffee addiction, so I can keep producing it for free 😸☕️

Buy me a coffee

References