3 ms·

SciPy stats includes a Wallenius’ discrete random variable[15], that takes four parameters on its creation: M is the total number of subjects, n is the number o

by Qem 3y ago

SciPy stats includes a Wallenius’ discrete random variable[15], that takes four parameters on its creation: M is the total number of subjects, n is the number of Type I subjects (journalists), and odds is the odds ratio: the odds of killing a journalist rather than a non-journalist when there is only one individual of each. The random variate represents the number of journalists that end dead if Israeli forces kills pre-determined x people from Gaza Strip one by one. The total population in the strip (M) is approximately known. x is known from OCHA daily stats. For the effective population of journalists (n) and odds ratio (odds) we can use Wallenius' to test several value pairs and check which best fit the data. From the article we can estimate the membership of the Palestinian Journalists Syndicate at 1200. That is a upper bound on the effective number of journalists, taking into account the likely inclusion of supporting staff and the many professionals forced to flee the Strip[16], sometimes after receiving explicit threats[17]. About the odds ratio, the current body count is about 1.3% of the total population in the Strip. Meanwhile, US estimates about 20%-30% of the 30000 strong Hamas forces were killed[18]. This number is probably exaggerated, as it implies every dead male in the age bracket around 13-60 is being counted as a Hamas militant, but for the sake of iron-manning the argument, let's assume those figures are true. That means the average Hamas militant is about 20:1 to 30:1 times more likely to be killed than a random person. So we should expect jurnalists' odds ratio to sit between somewhere between random citizen and Hamas militant, taking into account occupational hazards but not deliberate targeting. In doubt I took 100:1 as an upper bound, and wrote a model (source bellow). The best fit I got over the search grid had an effective journalist population of 155 and odds ratio of 75:1. As the data is noisy, I took the top 20 matches to have some idea of error intervals, and got ranges of 135 - 205 effective population and 51 - 96 odds ratio. The odds ratio is larger than our estimate for Hamas militants, indicating that in fact Israeli forces are busier targeting journalists than Hamas militants. Even if we assume the total kill counts are underestimates, given all people missing under the rubble and lack of proper rescue work due to targeting of rescue crews, say, adding a xs = [2*x for x in xs] line after the first definition of xs, to double the total counts, we still get killing odds for journalists equal or greater than for militants. [1]. https://en.wikipedia.org/wiki/Decimation_(punishment) https://en.wikipedia.org/wiki/Decimation_(punishment) [2]. https://xkcd.com/356/ https://xkcd.com/356/ [3]. https://www.reuters.com/world/middle-east/global-media-outlets-sign-letter-urging-protection-journalists-gaza-2024-03-01/ https://www.reuters.com/world/middle-east/global-media-outle... [4]. https://en.wikipedia.org/wiki/Urn_problem https://en.wikipedia.org/wiki/Urn_problem [5]. https://www.bbc.com/news/world-middle-east-68445973 https://www.bbc.com/news/world-middle-east-68445973 [6]. https://euromedmonitor.org/en/article/6202/Israeli-tanks-have-deliberately-run-over-dozens-of-Palestinian-civilians-alive https://euromedmonitor.org/en/article/6202/Israeli-tanks-hav... [7]. https://en.wikipedia.org/wiki/Noncentral_hypergeometric_distributions https://en.wikipedia.org/wiki/Noncentral_hypergeometric_dist... [8]. https://en.wikipedia.org/wiki/Wallenius%27_noncentral_hypergeometric_distribution https://en.wikipedia.org/wiki/Wallenius%27_noncentral_hyperg... [9]. https://en.wikipedia.org/wiki/Fisher%27s_noncentral_hypergeometric_distribution https://en.wikipedia.org/wiki/Fisher%27s_noncentral_hypergeo... [10]. https://cpj.org/2024/03/journalist-casualties-in-the-israel-gaza-conflict/ https://cpj.org/2024/03/journalist-casualties-in-the-israel-... [11]. https://twitter.com/EuroMedHR/status/1761121386347516095 https://twitter.com/EuroMedHR/status/1761121386347516095 [12]. https://en.wikipedia.org/wiki/List_of_journalists_killed_in_the_Israel%E2%80%93Hamas_war https://en.wikipedia.org/wiki/List_of_journalists_killed_in_... [13]. https://en.wikipedia.org/wiki/Al-Rashid_humanitarian_aid_incident https://en.wikipedia.org/wiki/Al-Rashid_humanitarian_aid_inc... [14]. https://www.ochaopt.org/ https://www.ochaopt.org/ [15]. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.nchypergeom_wallenius.html https://docs.scipy.org/doc/scipy/reference/generated/scipy.s... [16]. https://www.instagram.com/byplestia/p/Cz9MfelKk3x/ https://www.instagram.com/byplestia/p/Cz9MfelKk3x/ [17]. https://english.ahram.org.eg/News/516373.aspx https://english.ahram.org.eg/News/516373.aspx [18]. https://www.timesofisrael.com/us-said-to-believe-israel-killed-just-20-30-of-hamas-terrorists-in-gaza-fighting/ https://www.timesofisrael.com/us-said-to-believe-israel-kill... 2/3 (Continues)

Qem 3y ago
3/3 (Source code used for analysis) import csv from collections import Counter from datetime import date, timedelta from itertools import accumulate, islice from matplotlib import pyplot as plt from scipy.stats import nchypergeom_wallenius from statistics import linear_regression def batched(iterable, n): # batched('ABCDEFG', 3) --> ABC DEF G if n < 1: raise ValueError('n must be at least one') it = iter(iterable) while batch := tuple(islice(it, n)): yield batch def dateab(a, b): # Lists dates from day a to day b. i = 0 while (c := a + timedelta(days = i)) <= b: yield c.isoformat() i += 1 def reader(file): with open(file, newline='', encoding='utf-8') as csvfile: for row in csv.reader(csvfile, delimiter=';', quotechar='"'): yield row[0:2] def sum_absdiff(iter1, iter2): # Compute absolute differences sum between vectors, as error measure. return sum(abs(x - y) for x, y in zip(iter1, iter2)) days = list(dateab(date(2023, 10, 7), date(2024, 3, 1))) weeks = [e for e in batched(days, 7) if len(e) == 7] total_killings = [(d, int(n)) for d, n in islice(reader('gs_cummulative_total_killings.csv'), 1, None)] d = dict(total_killings) xs = list(max(d.setdefault(e, 0) for e in week) for week in weeks) journalist_killings = [t[0] for t in reader('gaza_strip_journalists_killed.csv')] c = Counter(journalist_killings) ys = list(accumulate(sum(c.setdefault(e, 0) for e in week) for week in weeks)) slope, intercept = linear_regression(xs, ys, proportional=True) fitted = [slope * x + intercept for x in xs] print('Linear regression parameters: slope =', slope, ';', 'intercept =', intercept, '\n') M = 2375259 # 2022 population estimate (source: https://en.wikipedia.org/wiki/Gaza_Strip) ns = [i for i in range(1, 1200) if i % 5 == 0] odds = [i for i in range(1, 100) if i % 3 == 0] candidate_store = {} for n in ns: for odd in odds: L = [] for x in xs: r = nchypergeom_wallenius(M, n, x, odd).mean() L.append(r) candidate_store[(n, odd)] = L[:] best_wallenius_fit = min(candidate_store, key= lambda e: sum_absdiff(candidate_store[e], ys)) print('n, odds (best Wallenius fit):', best_wallenius_fit, '\n') top_20 = sorted(candidate_store, key= lambda e: sum_absdiff(candidate_store[e], ys))[:20] print('Top 20 (n, odds):', top_20, '\n') top_20_ns = [e[0] for e in top_20] top_20_odds = [e[1] for e in top_20] print('Top 20 n variation in range:', min(top_20_ns), '-', max(top_20_ns)) print('Top 20 odds variation in range:', min(top_20_odds), '-', max(top_20_odds)) fig, ax1 = plt.subplots() ax1.plot(xs, ys, label='Available data') ax1.plot(xs, fitted, label='Linear regression fit') ax1.plot(xs, candidate_store[best_wallenius_fit], label='Wallenius\' distribution fit') ax1.set_xlabel('General population (source: ochaopt.org)') ax1.set_ylabel('Journalists (source: wikipedia.org)') ax1.set_title('Cummulative killings by Israeli forces in Gaza Strip\n2023-10-07 to 2024-03-01') ax1.legend() plt.show() Output (in addition to figure above): Python 3.11.8 (main, Feb 7 2024, 00:00:00) [GCC 13.2.1 20231011 (Red Hat 13.2.1-4)] on linux Type "help", "copyright", "credits" or "license()" for more information. ========== RESTART: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ========== Linear regression parameters: slope = 0.00344597608865967 ; intercept = 0.0 n, odds (best Wallenius fit): (155, 75) Top 20 (n, odds): [(155, 75), (160, 72), (145, 84), (165, 69), (170, 66), (150, 78), (175, 63), (140, 87), (135, 93), (150, 81), (185, 60), (140, 90), (180, 60), (190, 57), (195, 54), (200, 54), (135, 96), (180, 63), (205, 51), (185, 57)] Top 20 n variation in range: 135 - 205 Top 20 odds variation in range: 51 - 96

by Qem 3y ago

Qem 3y ago

3/3 (Source code used for analysis) import csv from collections import Counter from datetime import date, timedelta from itertools import accumulate, islice from matplotlib import pyplot as plt from scipy.stats import nchypergeom_wallenius from statistics import linear_regression def batched(iterable, n): # batched('ABCDEFG', 3) --> ABC DEF G if n < 1: raise ValueError('n must be at least one') it = iter(iterable) while batch := tuple(islice(it, n)): yield batch def dateab(a, b): # Lists dates from day a to day b. i = 0 while (c := a + timedelta(days = i)) <= b: yield c.isoformat() i += 1 def reader(file): with open(file, newline='', encoding='utf-8') as csvfile: for row in csv.reader(csvfile, delimiter=';', quotechar='"'): yield row[0:2] def sum_absdiff(iter1, iter2): # Compute absolute differences sum between vectors, as error measure. return sum(abs(x - y) for x, y in zip(iter1, iter2)) days = list(dateab(date(2023, 10, 7), date(2024, 3, 1))) weeks = [e for e in batched(days, 7) if len(e) == 7] total_killings = [(d, int(n)) for d, n in islice(reader('gs_cummulative_total_killings.csv'), 1, None)] d = dict(total_killings) xs = list(max(d.setdefault(e, 0) for e in week) for week in weeks) journalist_killings = [t[0] for t in reader('gaza_strip_journalists_killed.csv')] c = Counter(journalist_killings) ys = list(accumulate(sum(c.setdefault(e, 0) for e in week) for week in weeks)) slope, intercept = linear_regression(xs, ys, proportional=True) fitted = [slope * x + intercept for x in xs] print('Linear regression parameters: slope =', slope, ';', 'intercept =', intercept, '\n') M = 2375259 # 2022 population estimate (source: https://en.wikipedia.org/wiki/Gaza_Strip) ns = [i for i in range(1, 1200) if i % 5 == 0] odds = [i for i in range(1, 100) if i % 3 == 0] candidate_store = {} for n in ns: for odd in odds: L = [] for x in xs: r = nchypergeom_wallenius(M, n, x, odd).mean() L.append(r) candidate_store[(n, odd)] = L[:] best_wallenius_fit = min(candidate_store, key= lambda e: sum_absdiff(candidate_store[e], ys)) print('n, odds (best Wallenius fit):', best_wallenius_fit, '\n') top_20 = sorted(candidate_store, key= lambda e: sum_absdiff(candidate_store[e], ys))[:20] print('Top 20 (n, odds):', top_20, '\n') top_20_ns = [e[0] for e in top_20] top_20_odds = [e[1] for e in top_20] print('Top 20 n variation in range:', min(top_20_ns), '-', max(top_20_ns)) print('Top 20 odds variation in range:', min(top_20_odds), '-', max(top_20_odds)) fig, ax1 = plt.subplots() ax1.plot(xs, ys, label='Available data') ax1.plot(xs, fitted, label='Linear regression fit') ax1.plot(xs, candidate_store[best_wallenius_fit], label='Wallenius\' distribution fit') ax1.set_xlabel('General population (source: ochaopt.org)') ax1.set_ylabel('Journalists (source: wikipedia.org)') ax1.set_title('Cummulative killings by Israeli forces in Gaza Strip\n2023-10-07 to 2024-03-01') ax1.legend() plt.show() Output (in addition to figure above): Python 3.11.8 (main, Feb 7 2024, 00:00:00) [GCC 13.2.1 20231011 (Red Hat 13.2.1-4)] on linux Type "help", "copyright", "credits" or "license()" for more information. ========== RESTART: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ========== Linear regression parameters: slope = 0.00344597608865967 ; intercept = 0.0 n, odds (best Wallenius fit): (155, 75) Top 20 (n, odds): [(155, 75), (160, 72), (145, 84), (165, 69), (170, 66), (150, 78), (175, 63), (140, 87), (135, 93), (150, 81), (185, 60), (140, 90), (180, 60), (190, 57), (195, 54), (200, 54), (135, 96), (180, 63), (205, 51), (185, 57)] Top 20 n variation in range: 135 - 205 Top 20 odds variation in range: 51 - 96