Published: 25 September 2019
My parliamentary colleagues, Yaacob Ibrahim and Amrin
Ladies and gentlemen
1. First, let me thank RSIS for organising this conference on Foreign Interference Tactics and Countermeasures.
2. It is a very relevant topic all around the world, and here in Singapore. I am going to start my remarks by taking a step back and looking at the bigger picture.
3. What is foreign interference? What are the methods, what is happening now? Essentially, foreign interference is when a country, external agencies, or people try to shape the behaviour, actions, policies of a target country.
4. This sort of foreign interference is age old, and it is a basic principle of International Relations.
5. We can go find examples in China for many of these things. During the Warring States Period [300 BC], we have the State of Yan - it is a very well-known example. It was led by a brilliant General, Yue Yi, and he conquered most of the state of Qi by 284 BC. But two well-fortified cities held up, Jimo and Ju.
6. King Zhao of Yan died 5 years later in 279 BC. The opposing general saw an opportunity – that maybe he could bribe Yan officials and pass on the rumour that General Yue Yi is plotting to take over the Kingdom of Qi – to create discord between the newly installed King and his most effective general.
7. Creating suspicion worked. General Yue Yi was dismissed, and was forced to flee. Yan’s forces were weakened and they were eventually driven away.
8. And we move over to the superpower around the 2nd Century BC - Rome. If we look at Rome and Greece in those times - Greek politicians were constantly quarreling. They began to appeal to Roman politicians and authorities, to gain in both intra-polis conflicts and inter-polis conflicts. They did that to get one-up to demolish their Greek opponents, both intra-polis and inter-polis.
9. To court favour with the superpower, Rome, some of the Achaean politicians also started to undermine the Achaean League’s unity. They lobbied Roman senators to intervene in Greek affairs, and prop-up pro-Roman collaborators in Greek states, 2000 plus years ago. Sounds familiar? There is a more recent example. History always repeats itself.
10. When Sparta wanted to secede from the Achaean League, pro-Roman embassies appealed to Rome to intervene. Over time, the independence of the Greeks was diluted. Rome was of course very happy to do all of these. The collective resolve of the Greeks weakened. This eventually led to the conquest of Greece.
11. This analogy brings to mind something that happened last week. The world saw activists from Hong Kong appearing before the US Senators, asking them to intervene in Hong Kong.
12. One difference though from the Roman/Greek situation and the US/Hong Kong situation is that Hong Kong has been given Special Economic status and treatment by the US. So, the US will say all that it is doing is that its Congress was considering whether Hong Kong has that degree of autonomy that qualifies it to continue to receive the Special Economic Treatment. And that it is entirely the prerogative of the US Congress to consider the matter.
13. But if you look at it from China’s perspective, Hong Kong is part of China. And even if one accepts that the US is entitled to look at Hong Kong’s position; that the US Congress is entitled to consider this - nevertheless, the appearance of Hong Kong activists in the US Congress, appealing to the US to intervene in HK - you can imagine how that will look from China’s perspective.
14. Active interference in another country’s affairs is a given in international relations. Collaborators within a country working with foreign interests is also a given, knowingly as well as unknowingly.
15. And if you look at methods of such interference, it has taken a variety of forms.
16. To start with, at the highest level, we have diplomatic channels. Often legitimate, because you use diplomacy to bring across viewpoints, and persuade other countries. All states engage in – we do it, everybody does it. Entirely legitimate.
17. But of course, these channels can also be used to subvert and interfere with other states.
18. And frankly, there are no angels in this business. So let’s drop the hypocrisy. Many big countries do this to smaller countries, and to one another.
19. We have been the subject of such favours from China, from Russia, from the US and the UK. That’s why I say there are no angels in this. During our struggles with the Communists from the 1950s to 1970s - the Communists in Malaya and Singapore received support from China and USSR – material, propaganda, overt, covert.
20. And that kind of interference is not restricted to that period of Communist insurgency. The US – again, I’ll just give one well-publicised example - in the late 1980s, the First Secretary at the US Embassy in Singapore, who went by the name of Hank Hendrickson, cultivated and encouraged a group of lawyers and told them to go into opposition politics. One of them was also offered refuge in the US, should things not go well. Things did not go well, and that gentleman did eventually get refuge in the US – political asylum and finally, citizenship.
21. It’s one thing to link up with politicians of all shapes and partisan views - Diplomats are entitled to do that. It’s quite another to try and set up political organisations, encourage citizens to take part in political processes, compete in elections, and offer funding and asylum. That, I think, crosses well beyond the bounds of normal diplomatic activity. But it was done.
22. The UK – in 1961, after we got self-governance, tensions reached a height within the ruling party, between the pro-communists and the non-communists. Those on one side of the divide, the pro-communists - Lim Chin Siong, Fong Swee Suan, Woodhull, James Puthucheary, were seeking to overthrow the elected government headed by Lee Kuan Yew. So they decided to meet the British Commissioner, Lord Selkirk. They wanted assurances that the British would not restore direct rule, if that happened, and Lord Selkirk gave them the assurances, which then encouraged them to bid for power on their own. This has been made famous, or infamous, as the Eden Hall Tea Party in Mr Lee Kuan Yew’s book.
23. Second, using covert agents of influence, under the control of intelligence agencies. You have recent reports from Australia, New Zealand. From Australia, there have been reports of attempts to control a senator, fund him, control him.
24. Ourselves - just two years ago, we expelled Huang Jing, an agent of influence. He was a professor at the Lee Kuan Yew School of Public Policy, and was in contact with foreign intelligence organisations and agents. He used his position at the school to engage prominent and influential Singaporeans. He told them he had ‘privileged information’. He tried to influence senior public officials. He tried to change and manage Singapore’s foreign policy. He also recruited others to help him.
25. Third, is the media. A key node - let’s be frank - through which foreign states can exert influence over domestic public opinion and in some cases, through the secret funding and control of publications. In other cases, having agents use the cover of journalists themselves.
26. It’s been used right through, if you go back to the Cold War. Both sides did this. The CIA secretly funded and controlled magazines and writers. One example was the Congress for Cultural Freedom - that was a conduit. It was a covert propaganda front to sway public opinion. The co-founder of an influential English language, Paris-based publication, The Paris Review, was a CIA agent.
27. The Soviets of course, also manipulated journalists and publications to push disinformation. Many of us would have heard the infamous claim that the AIDS virus was developed in a US military facility. False – it was put out by the Soviets. It first appeared in an Indian newspaper, The Patriot, which was set up by the KGB in the 1960s. These things happened all the time.
28. In the 1970s in Singapore, we had two such operations involving our newspapers - The Eastern Sun and The Singapore Herald. The Eastern Sun worked with a news agency of Communist China and received funding from them. The Singapore Herald took money from foreign sources - a Malaysian politician. It pushed an anti-Government line and was also stridently against National Service, which was a key pillar in defending Singapore. The Singapore Herald continuously ran articles against National Service and took money from a Malaysian politician.
29. More recently, we have received reports again from Australia and New Zealand of newspapers, which receive money from overseas, and push foreign countries’ viewpoints.
30. Another way of doing this, NGOs. Again, there are many, many examples. The Ford Foundation has been cited by academics as having been a conduit for money to be given to anti-Communist causes.
31. Another example is this. States have been known to target cause-based movements in other states. They mobilise activists to advance foreign countries’ interests.
32. The Soviets used this very well to boost their anti-nuclear movement in Europe during the late 1970s and early 1980s to weaken the Western military alliance. It was particularly effective in galvanising Western public opposition to NATO’s deployment of the Neutron Bomb. That pressure even eventually led to President Carter deciding to cancel its deployment of the Neutron Bomb.
33. So, this brief survey would show that neither the desire to intervene, to shape and manipulate a target country, nor the methods, are limited. It is age-old and part and parcel of how countries exercise their power. It is not new. But the whole concept has been, in a sense, turbo-charged and revolutionised because the Internet has opened up limitless possibilities to advance these interests.
34. As can be expected, there is a military doctrine developed for the Internet age – it is called the Gerasimov Doctrine, named after the Russian Military Chief of Staff. He has said that The "Rules of War" have been redefined, such as using non-kinetic military measures such as Hostile Information Campaigns (HICs). What they can do is to identify what they call the "protest potential" of any population of a target country, then create protests, deepen divisions and increase hostility among the different groups, and get them to distrust institutions. In that country, trust in institutions and systems gets damaged, and the people lose faith in democracy as a whole.
35. So if you look at the surveys - the number of people who prefer military rule, whether in the US or in Western Europe, is increasing. There are an increasing number of people who are losing faith in traditional political parties, and being attracted to strongmen and specific populist ideologies. Part of it is because trust has broken down, and not simply because of HICs. Trust has broken down for a variety of reasons, including growing inequality, the wealth gap, globalisation and the consequences of countries not catering effectively to people who have been left out by globalisation. Trust has been broken down for a variety of reasons. People see that the system benefits a small group, and they are not being taken care of. That is being effectively exploited by HICs as well.
36. The Gerasimov Doctrine states that these non-kinetic measures, done through the Internet, can in many cases exceed the power of force and weapons. And you don’t need conventional warfare. You exploit the protest potential, keep the population in the country in a constant state of turmoil and ineffectiveness and degrade their ability to deal with it – including their own economic issues, or external threats. That is how you bring down a country.
37. Wars no longer need to be declared because the internal opposition is created as a "permanently operating front" in the target country.
38. The fault-lines are also exploited by bad actors, both internal and external, on hot-button issues. They tap on legitimate sentiments, they target reasonable people. They use legitimate news outlets as conduits. They convert disinformation into mainstream information, they enlist what Lenin famously called "useful idiots" to the cause.
39. One example is the Nancy Pelosi ‘deepfake’. It should actually more accurately be called a ‘cheapfake’ because it is so primitive. It was spread on social media, picked up and given some legitimacy because some mainstream media reported on it.
40. In other cases, bad actors instigated protests and counter-protests, and infiltrated legitimate activist groups on both sides to create and exploit the possibilities of hot button issues. For example, in the US, you have "Blue Lives Matter", an organisation that has been set up to support police officers in response to the movement "Black Lives Matter". And a foreign agency was said to be on both sides trying to create riots.
41. The Internet has made HICs cheap, easy and effective to mount. There is a growing commercial industry which supports all of this. Last year, the Parliamentary Select Committee (on Deliberate Online Falsehoods) was given some of the going rates for tools and services – you want one million Instagram "likes" – only US$18. You want 100 Twitter followers, likes or re-tweets – 34 US cents. You want 100 YouTube subscribers – 66 US cents. You want to use online propaganda to instigate a street protest in the US – US$200,000. You can do it.
42. Princeton University did a count of large-scale online "foreign influence efforts" from 2013 to 2018. They found 53 distinct instances, targeting 24 different countries. By no means is this comprehensive.
43. Some of these attempts have been targeted squarely at influencing the outcome of elections, or national referendums. The famous example, in a way, that started a lot of the discussions, is of course the 2016 US Presidential Elections. A foreign troll factory was said to have conducted a disinformation campaign using 50,000 bot accounts, over 3,800 fake Twitter accounts, and nearly 500 fake Facebook accounts. Facebook’s own best estimate is that from 2015 to 2017, approximately 126 million people may have received content from this troll factory and its associated accounts.
44. The combination of these online HICs, and the offline activities - foreign controlled media / sites, agents of influence, NGOs, groups of citizens who fan the flames, knowingly and unknowingly - all of this combined is extremely toxic, extremely powerful.
45. Ukraine is a recent example. Mr Ruslan Deynychenko (co-founder of StopFake.org) gave evidence to the Parliamentary Select Committee on Deliberate Online Falsehoods last year. He said that a foreign country had built a narrative against Ukraine, that the Ukrainian government was fascist and corrupt. A combination of online and offline methods were used to distribute and amplify these narratives.
46. The foreign country’s proxies indirectly owned Ukrainian television networks and media-holding companies. They used them to exert influence over Ukrainian media. This was coupled with the spread of television and radio signals into Ukrainian territory to increase the reach of propaganda, including political "talk shows". Who took part in these political talk shows? Hired paid actors, and these chaps posed as "experts" and "representatives" of Ukraine. They said various things that needed to be said, to create the appearance of fair and balanced reporting.
47. There was also the financing of right-wing extremist groups to create provocations, like burning flags, and desecrating monuments and military cemeteries. And spreading online disinformation within social networks in Ukraine, with networks of "bots" calling for massive anti-Government riots, including false news of atrocities such as young girls being raped or children being crucified. Nothing is off limits.
48. The Ukraine Crisis Media Centre says the foreign country also made use of "useful idiots", enlisting them as opinion leaders, academics, think tanks, politicians and community leaders, to advance its narrative.
49. The efforts were so successful not just in Ukraine, but also the rest of Western Europe. When the Dutch were deciding whether to support Ukraine for a specific foreign policy initiative, a majority, out of a small turnout of citizens, voted against. Dutch public opinion was manipulated successfully and effectively. Likewise, Czech public opinion was also effectively manipulated to make the citizens believe that Ukraine was fascist and corrupt and should not be supported.
50. I will end off this point of the speech by quoting James Comey, former FBI Director, who told the House Intelligence Committee two years ago, in March 2017, and I quote: "One of the things we radiate to the world is the importance of our wonderful, often messy, but free and fair democratic system and the elections that undergird it. And so when there’s something by a foreign nation state to mess with that, to destroy that, to corrupt that, it is very, very, serious and threatens what is America". The very idea of America.
51. A survey of what is happening in Western Europe will show that something similar is happening in many countries. If you look at Brexit - surveys by King’s College London suggested that 40% of Brits still believe that Britain sends 350 million pounds a week to the European Union. Nearly a third still believe that areas of Britain are controlled by Sharia law, and there are no-go areas for non-Muslims. And these people overwhelmingly voted for Brexit, and there was a marginal victory for Brexit, about 4%.
52. So, it’s tremendous. Mainstream media was also liable – they said that 12 million Turks were going to go into the UK, and so on. The online falsehoods and negative campaigning online, which played on anti-Muslim sentiment and xenophobia, was very effective.
53. All of that hasn’t happened in its full glory in Singapore, but it can. Some of it has already happened. And we also see some nascent attempts to combine the different approaches. I will give you one example.
54. A group of activists met with Dr Mahatir, the Malaysian Prime Minister last year. PJ Thum urged him to bring democracy to Singapore; amongst other countries. PJ Thum also said Singapore should become part of Malaysia, and celebrate Independence on 16 September - Malaysia Day. Thum and his partner Kirsten Han, who also met Dr Mahathir, set up an organisation called New Naratif – which is significantly funded by a foreign foundation and received other foreign contributions as well.
55. Ms Han, on video, has said that Singapore has failed compared to Hong Kong, because 500,000 people don’t go on the streets to march, unlike Hong Kong. She wants to change that, through classes run by New Naratif.
56. This will seem ridiculous on so many levels, but we can leave that aside because everyone is entitled to their views, however reasonable or unreasonable. My primary point is that is it right for foreign funding to be received in order to advance these viewpoints? That’s the question that should be posed. These are matters that Singaporeans can argue about – do we think it is appropriate for 500,000 people to go on the streets and you want to run classes on that? Well, that’s one view. But should there be foreign funding? Should foreign NGOs be involved? Should foreign donations be received in order to push these lines? She’s on video saying it.
57. There is The Online Citizen (TOC), an online news site which targets Singaporeans. TOC uses foreigners, employs them, including Malaysians, to write almost exclusively negative articles on Singaporean social and political matters, including inflammatory articles that seek to fracture social cohesion. They supported a call for Singaporean civil servants to follow the example of Hong Kong civil servants in protesting. They made allegations about the Prime Minister, which has led to a civil suit by the Prime Minister, that they have falsely attacked his character and fitness to hold office. These two articles are by a Malaysian, who, based on publicly available information, is said to be in Shah Alam, near Kuala Lumpur - a lady by the name of Rubaashini.
58. I am not commenting on the legal merits of the article, since it is the subject of a lawsuit. Only that a foreigner, staying in Malaysia, wrote these things, for a Singapore site, to target a Singapore audience. Telling Singapore civil servants to protest and calling into question the Prime Minister’s integrity. She has written many other articles to try and influence viewpoints within Singapore. Who controls her? Who pays her? What is her purpose? These are all legitimate questions. Appearing on the Internet, on TOC, most readers would just assume that this was by a genuine Singaporean contributor.
59. There are many other Malaysian writers as well on TOC. It is said that for TOC, out of 14 admins, only five are located within Singapore. Nine are outside. Four in Malaysia and two in Indonesia. We don’t know who they are. Are they Singaporeans? Are they foreigners?
60. There is a grey area here. For responsible media, both Singaporean and those employing foreigners, there would be the assumption that they will have some ethics. Of course that can be exploited, but they are subject to a framework. In every country, there is a framework for how media behave.
61. For online news sites - as we can see from what has happened in Europe and the US and in many other countries, including Asia, there are anonymous writers where no one knows who they are. Their motivations and who is paying them is unknown. For all you know, they could be foreigners, as we see in the case of TOC. Writing inflammatory stuff and having no interest in social and political stability within the country. Their only interest is to get eyeballs and if they are under the influence of other agencies, they can easily be used as tools for foreign interests. Such sites have been used by foreign countries to attack and deepen divisions.
62. This is just a small snapshot example of how viewpoints can be manipulated by foreigners. Conflicts, riots, arguments – these all have been organised elsewhere and can be done here.
63. So what should the response from countries be? Some, in particular tech companies, suggest self-regulation.
64. I know there are representatives from tech companies here, and we have been having conversations with them for the last two years or so, but we are not the only ones. Many countries are having conversations with them.
65. The question is, can tech companies be left to self-regulate, in the absence of legislation? I think the clear answer is No.
66. For a start, let’s see the responses from the tech companies so far. I think the responses have been varied and there have been challenges. From denial that there are problems, to taking some reasonably effective steps.
67. Part of the issue is that their business model militates against proper self-regulation. The more users, the more content there is on their platforms, the more user attention they can sell to advertisers, the more their profits.
68. Removing fake users, removing fake accounts, investigating into coordinated inauthentic behaviour - these are all costly. The tech companies are in a position of conflict where their business interests often conflict with what needs to be done in the broader society’s interests.
69. You look at Company Law; at how companies are regulated. You look at how conflicts of interest are resolved. No one says "you decide for yourself and you do it." Within a company, the company sets the framework, and there will be other officers who will then look at the conflict situation and decide what needs to be done.
70. Within countries, there will be laws that deal with how in specific industries conflicts of interest ought to be resolved. It cannot be any different for tech companies. There is no difference in principle as to why they should be different.
71. We as a Government would like to work with the tech companies. Tech companies are our partners, they are not our opponents.
72. Mr Mark Zuckerberg himself has said in March this year that regulation is necessary. And that this is beyond tech companies. But he also says there needs to be global standards agreed to by all the countries for such legislation.
73. As one Canadian newspaper put it, of course, when will there be global agreement on the standards to be applied for a legislation that all countries can apply. The newspaper said on the "10th of Never", which means never. Would you expect US, Russia, China, for a start, to agree together on common standards for what is not acceptable? And what the common standards ought to be? Suggestions that there can be legislation are welcome, but suggestions that such legislation should be based on universal common standards, I think are not very practical. The different social, political and cultural contexts in each country will make a broad international agreement nearly impossible.
74. Fundamentally, and this is my main point – let’s be clear, what are we talking about here? It is not about specific commercial interests, it is not about specific arguments. Ask ourselves a number of questions.
75. First of all, does it happen? Do countries target other countries’ using Internet platforms? The answer is clear.
76. Second, has it been shown to operate and does it have a serious impact on other countries? The answer is obvious – yes.
77. Does it have serious national security implications? You look at the US, you look at France presidential elections, you look at UK, you look at other countries in Western Europe, and you look at other countries in Asia.
78. You look at the riots in Sri Lanka last year where Facebook refused to take down statements and positions by Sinhalese calling for killing of Muslims, and people died. My colleague Edwin Tong asked Facebook in London during the Grand Committee hearing – Facebook said yes, it was a mistake. But you know, people paid with their lives for such mistakes. At that time Facebook refused take it down because they said it did not contravene their standards.
79. So is it for Sri Lanka to decide its national interest? And is it a sovereign right to protect itself? Or is it for a private sector company to decide these issues? I think the answer, if you put out questions in those terms, is unarguable. Every country has the sovereign right to decide for itself how it will protect our national security interests. Even if I wanted to, I cannot give that away. If I give that away, it is an abdication of responsibility.
80. And the Government with the consent of the people will have to decide. Commercial interests cannot tell us what to do about this. They have to work within the framework of the law. Our task is to make sure that the approach is fair and reasonable, and at the same time effective.
81. The state cannot take a hands-off approach. The serious impact of HICs on the social fabric, on political sovereignty, on peace, on stability, and on national security, has to be met head on. And it has to be met head-on by states, working with tech companies as partners.
82. I think it is useful to look at what some countries have done.
83. France has introduced an Information Manipulation Law. The law mandates transparency over social media platforms’ algorithms and election advertising. It allows the French national broadcasting agency to suspend television channels "controlled by a foreign state or under the influence" of that state if they deliberately disseminate false information likely to affect the integrity of the elections.
84. In Germany you have the Network Enforcement Act, which was also strongly opposed by the tech platforms. It compels social networks to monitor and remove illegal online content. And I quote, "obviously illegal" hate speech and other postings must be removed within 24 hours of receiving a notification, or the platforms may face fines.
85. Australia passed a package of new laws very quickly in 2018 which were aimed at preventing foreign interference. It includes restrictions on foreigners making political donations, stronger espionage laws, tougher penalties, and a requirement that agents or lobbyists who represent foreign nations or entities must register their interests. Political entities and campaigners cannot receive $1,000 or more from a foreign donor.
86. There are registration obligations imposed on persons or entities who have arrangements with or undertake certain activities, for example, lobbying on behalf of foreign principals. New foreign interference offences have been created, targeting covert, deceptive, or threatening actions by foreign actors who intend to influence Australia’s democratic or Government processes, or to harm Australia.
87. Israel has put in transparency requirements for NGOs receiving more than half of their funding from foreign state sources.
88. We put in legislation – the Protection from Online Falsehoods and Manipulation Act (POFMA) that deals with falsehoods. That’s a framework for encouraging discussion based on facts. It allows corrections to be carried, requires corrections to be carried, if there are falsehoods which affect public interest. But it doesn’t deal with HICs. It wasn’t intended to deal with HICs, because well-done HICs don’t just depend on falsehoods - it would be an entire apparatus targeting a target country using a mixture of falsehoods. It’s like if you put spies in, they don’t necessarily engage in spying right from the beginning.
89. If the target country knows that such and such a person is under the control of a foreign agency, every country has rules and laws that allows their arrest. In Singapore, we use the Internal Security Act to detain such people.
90. HICs are a massive thing and POFMA was not meant to deal with them.
91. There are two other elements to be dealt with. One is hate speech which targets racial and religious sensitivities, part of it is being taken care of under the Maintenance of Religious Harmony Bill. It has been introduced in Parliament, and the Second Reading will be next month in October. It deals with hate speech targeting religious divides and sensitivities. The Act has been around for nearly 30 years. This updates the legislation to take into account the changes in the environment over the last 30 years or so.
92. I have said separately we will put in legislation to deal with HICs, and I think we need to do that. If you look at what powers would be necessary to counter foreign interference, which includes HICs, it will have to give the Government powers to make targeted, surgical interventions, to investigate and respond expeditiously to HICs. Which also means getting the information so that we are able to investigate the provenance of content to see whether, to what extent it is foreign interference and to have the appropriate response.
93. France’s law targets falsehoods during elections. But as I just explained, HICs use a range of content, not just falsehoods, and it’s not usually just restricted to just election periods.
94. HICs, as the Gerasimov Doctrine makes clear, is where you constantly keep the other society off-balance by increasing and exploiting the "protest potential" as it is called.
95. There is also evidence on how a foreign power helped the rise of the Far Right in Germany, and how that rise of Far Right has materialised in Germany. These are serious matters.
96. So, the legislation needs to be able to deal with this diverse range of threats, including the flow of funds. And we may also need to consider how we restrict foreign participation in the leadership of specific organisations, and say Singaporeans are fine, but to what extent should foreigners be there? They are closely involved in our political landscape.
97. This is similar to our position on foreign participation in cause-based public assemblies and processions.
98. As I sum up, I would like to leave you two thoughts: First, foreign interference is an age-old threat, which has adapted to modern technology, and states must be able to deal with these threats.
99. Second, this is an issue of sovereignty and national security. The Government has to lead from the front, and we need to ensure we have the right tools to fight this threat. International cooperation will obviously be necessary, but I am not holding my breath that a proper international cooperation with acceptable standards will materialise very quickly. But we hope that it will come through at some point in time.
100. I look forward to a fruitful and in-depth discussion on this issue today. And I thank the panellists for coming here to share with us your expertise and experience, and I thank the organisers for organising this session. Thank you very much.