The internet is more than an enabling technology: it also enables criminals and terrorists to do evil in ways that were never before possible, explains Sir David Omand.

The internet presents a range of challenges and threats that are new – for the individual, for businesses and for governments. Let us consider human activity today on the internet.

The first layer is our everyday activity on the internet. Communicating, sharing, entertaining, trading. That activity is under constant attack from cyber criminals. We need very secure encryption in everyday communications to protect our privacy, our intellectual property and to defeat the cyber criminals.

Retaining confidence in the internet and its financial systems and transactions is fundamental for our economic wellbeing, which is why the British Government is spending £860m on major cyber security programmes. Data Protection legislation, both national and European, tries to protect our personal data from unauthorized use.

But there are others communicating via the internet who mean us harm. These are the dictators, terrorists, insurgents, proliferators, narcotics gangs, criminal groups and people traffickers, not to mention the Russian paramilitaries in Ukraine and ISIL jihadists in Iraq and Syria. In the dark web, beyond the indexing of Google, jihadist beheading videos are circulated and counterfeit goods, malware, drugs, sex and slaves are sold.

Underneath, supporting this everyday activity and trying to police the worst abuses on the internet, is a law enforcement layer. To protect us the police have the right to obtain information about the patterns of communications of suspects, terrorists and criminals of all sorts, under conditions that we in society legislate for and oversee. When necessary, the police have the right to seek warrants to access the content of those communications. Whether it is trying to locate a missing schoolgirl, test an alibi, or uncover a terrorist assassination plot, access to communications data is the most important investigative tool the police say they have. The Home Secretary has said that communications data has played a significant role in every MI5 counter-terrorism operation over the last decade. It has been used as evidence in 95% of all serious organised crime cases handled by the Crown Prosecution Service.

Of course, the publicity around Edward Snowden has highlighted tension between our right to everyday privacy at one level and the inevitable intrusive nature of detective investigation at the other. But this is resolvable since security and the protection of rights go together in a democracy.

The European Convention on Human Rights has always recognized that a balancing act exists between human rights and personal privacy on the one hand and the right to protection on the other. But this is provided that a set of conditions are in place: powers must be set down in legislation, the principles of proportionality and necessity followed, there is right authority and an audit trail, independent oversight, independent adjudication of claims of abuse and so on.

The UK has all of that – and I believe our system is the European model to follow. That is not to be complacent that oversight cannot be improved in its details – we should always seek improvement, but on the basis of the model we already have.

At the law enforcement level, there is regulated international co-operation, for example through advance passenger information and watch list data exchanges and liaison on suspects through Interpol and Europol. But there are growing and serious problems for law enforcement.

• The rapidity of the growth of crime on the internet is running well ahead of the capabilities of law enforcement, with criminals using internet technology directly through malware and also simply as a more efficient way to conduct traditional crimes such as fraud, but at scale.
• The tools or exploits for cyber crime can be bought from hacking specialists, so those conducting cyber crime do not now need to be software hackers themselves.
• The most serious cyber criminals are based in jurisdictions overseas where Mutual Legal Assistance requests and European arrest warrants may not be respected.
• The advent of digital technology is making the task for law enforcers of obtaining communications data and warranted communications much harder.
• Even where they wish to co-operate, the traditional telecoms and cable companies are increasingly physically unable to respond to legal warrants and provide the information to which the authorities are legally entitled since they have no business need to collect or retain information about their customers’ use of digital services that are free at the point of use and are covered by the flat rate subscription.
• Many of the modern Internet Service Providers (ISPs) are located overseas. The US Internet Service Providers, for example, apply their own company judgments whether to provide the British police with communications data on a suspect. The EU Data Retention directive aimed at easing this situation was recently found to be legally faulty, and the UK has had to legislate in the Data Retention and Investigatory Powers (DRIP) Act to put the UK legal position – which does have the necessary legal safeguards – beyond doubt.
• Another area of difficulty is gaining access to encrypted material on suspects’ computers and mobile devices on which vital evidence may lie. For example, in prosecuting members of paedophile networks. For example, Apple recently announced that to give more confidence to its customers it had deliberately written its IOS8 software to be so secure that the company itself cannot unlock mobile devices protected with a full password. But that means that when the police present a legal warrant authorizing examination of, say, a terrorist’s or kidnapper’s laptop, the company will deliberately have put itself in a position of being unable to help. Not what I would call corporate social responsibility.
• Finally, there is the sheer diversity of means of hiding communications using social media or on-line video games. Again, Snowden has done us a disservice by publicizing this.

This range of challenges has required intelligence services to adjust their operations in order to respond effectively.