Seit kurzem befasse ich mich mit Sprachenerkennung. Nein, ich meine nicht speech recognition, sondern language detection, also das automatische Erkennen der Sprache eines Textes.

Eigentlich wollte ich es nicht schon wieder selber machen, und suchte im Netz nach einer vorhandenen Möglichkeit, Texte zu Sprachen zuzuordnen. Doch leider konnte ich keine vorgefertigte LGPL Open Source Java API dafür finden.

Eine Möglichkeit, die ich jedoch noch nicht in Betracht gezogen habe, ist die Classifier4j API, welche mit Bayesschen Netzen und Vector-Space Modellen aufwartet. Aber damit beginnt schon die Komplexitätsbetrachtung einer Lösung.

Sicherlich gibt es recht komplexe Klassifikationsansätze, doch meine derzeitige Arbeit beschränkt sich auf einfache Methoden, auf die mich zwei Erinnerungen und ein Dokument brachten.

Die erste Erinnerung bezieht sich auf ein Thema, mit dem ich mich früher gerne befasst habe: das Thema Kryptologie, wo man für verschiedene Sprachen die Häufigkeiten der verschiedenen Buchstaben ermittelt, um bei monoalphabetischen Substitiutionen den Brute-Force Angriff in einen statistischen Angriff umzuwandeln. Damit ist klar, dass jede Sprache eine andere Häufigkeitsverteilung der Buchstaben besitzen sollte.

Die zweite Erinnerung bezieht sich auf etwas, dass mir mein Kommilitone Roman mal erzählte: In ihrem Mathematik-Praktikum sollten sie Texte anhand von statistischen Analysen, also Häufigkeiten von Buchsten und Buchstabenkombinationen, das Fachgebiet des Textes ermitteln.

Schließlich fand ich ein Studentenprojekt im Internet mit dem Titel ToSoBaP – Towards Solving the Babel Problem, was sich genau mit diesem Thema beschäftigt. Doch obwohl einige Teile der dort beschriebenen API scheinbar bereits realisiert wurden, hat man keinen Zugriff darauf. Dafür ist dort eine sehr interessante Spezifikation zu finden, welche die Grundideen zusammenfaßt.

Dort heißt es zur Sammlung von Methoden (S. 7):

Statistische:
  - N-gramme (Uni-/Bi-/Tri-) für
    - Zeichen
    - Konsonant-Vokal Folgen
  - Häufigste (also kurze) Wörter
  - HMMs: Zeichen und Übergangswahrscheinlichkeiten
  - Durchschnittliche Wortlänge

Eher linguistische:
  - Charakteristische Wörter, z.B. grammatische
  - Silbenbau
  - Morphologie: Prä- und Suffixe
  - Zeichensatz, Sonderzeichen
  - Textstruktur (Zitate, Absätze)

Eine schöne Zusammenfassung. Die eher komplizierteren Varianten Silbenbau, Textstruktur, HMM (Hidden Markiv Modelle, wie etwa Bayessche Netze) seien erst einmal auf einen sehr viel späteren Zeitpunkt zurückgestellt, da sie doch recht kompliziert sind.
Demgegen sind die anderen Ansätze recht simpel zu implementieren.

N-gramme

N-Gramme sind einfach Kombinationen aus Buchstaben. Das N steht für die Länge der Kombinationen. Die einfachste Variante sind die Unigramme, also einfach nur die einzelnen Buchstaben. Ich habe für meine kleine Anwendung eine Liste von analysierten Buchstaben aufgestellt, zumal ich eigentlich nur Deutsch und Englisch beherrsche und mich mit chinesischen Symbolen nicht herumschlagen wollte. Natürlich schließt dies automatisch die Erkennung aller nicht-Latein-Lettern benutzenden Sprachen aus, aber mir soll es vorerst reichen. Man kann das ganze natürlich auf den gesammten UTF-8 Zeichensatz ausbauen, aber damit steigt auch der Speicher und Zeitbedarf.

Also: LETTERS = “abcdefghijklmnopqrstuvwxyzäöüïßABCDEFGHIJKLMNOPQRSTUVWXYZ”

Wie man sehen kann, habe ich zwischen Groß- und Kleinbuchstaben unterschieden. Ist doch offensichtlich, dass die Deutsche Sprache mit großgeschriebenen Nomen mehr Großbuchstaben enthält als die Englische Sprache, welche nur Namen und am Satzanfang Kapitallettern benutzt.
hat natürlich den nachteil, dass man in deutschen texten, in denen der author zu faul war, die groß-/kleinschreibung anzuwenden, gefahr läuft, die Spache falsch zu klassfizieren…

Mittels einer Hash-Tabelle werden die absoluten Häufigkeiten verschiedener Texte in Deutsch und Englisch ermittelt. Die Texte stammen übrigens aus dem Projekt Gutenberg.

Das Ergebnis für die Unigramme aus 16 deutschen und 16 englischen Texten:

analyserLetterFrequencyAnalyser
languagedeutsch
total5028264
tokenabcdefg
abs.245.423083.793151.053246.300854.427065.149142.384
rel.0,048810,016660,030040,048980,169920,012960,02832
tokenhijklmn
abs.233.490415.379006.435050.511178.153109.049518.658
rel.0,046440,082610,001280,010050,035430,021690,10315
tokenopqrstu
abs.103.733021.512000.464334.458319.406284.869182.984
rel.0,020630,004280,000090,066520,063520,056650,03639
tokenvwxyzäö
abs.030.029070.533001.236002.726051.973024.918012.483
rel.0,005970,014030,000250,000540,010340,004960,00248
tokenü�ABCDE
abs.027.973023.995018.552017.513000.893016.434016.564
rel.0,005560,004770,003690,003480,000180,003270,00329
tokenFGHIJKL
abs.009.099017.998007.689008.295002.578008.308006.778
rel.0,001810,003580,001530,001650,000510,001650,00135
tokenMNOPQRS
abs.011.180005.879002.506007.192000.225007.580023.116
rel.0,002220,001170,000500,001430,000040,001510,00460
tokenTUVWXYZ
abs.006.439006.144012.157015.938000.025000.029005.657
rel.0,001280,001220,002420,003170,000000,000010,00113

analyserLetterFrequencyAnalyser
languageenglish
total5043940
tokenabcdefg
abs.394.993069.709120.655216.000629.813113.613095.681
rel.0,078310,013820,023920,042820,124870,022520,01897
tokenhijklmn
abs.305.292322.091004.294037.988194.824127.063352.083
rel.0,060530,063860,000850,007530,038630,025190,06980
tokenopqrstu
abs.385.132082.140004.927290.289305.490447.824141.722
rel.0,076360,016280,000980,057550,060570,088780,02810
tokenvwxyzäö
abs.047.739111.964008.419095.931002.976000.000000.000
rel.0,009460,022200,001670,019020,000590,000000,00000
tokenü�ABCDE
abs.000.000000.000008.515006.247005.186004.156003.705
rel.0,000000,000000,001690,001240,001030,000820,00073
tokenFGHIJKL
abs.002.583002.377010.264026.620003.108000.820003.845
rel.0,000510,000470,002030,005280,000620,000160,00076
tokenMNOPQRS
abs.008.657003.868002.943003.237000.155002.617008.010
rel.0,001720,000770,000580,000640,000030,000520,00159
tokenTUVWXYZ
abs.016.233000.697000.818007.122000.411003.053000.041
rel.0,003220,000140,000160,001410,000080,000610,00001

Zusätzlich zu den Unigrammen habe ich auch die Bigrammverteilungen ermittelt. Bigramme sind, wie der Name schon sagt, alle Kombinationen aus zwei Buchstaben, wie etwa aa ab ba ac ca ad…
Man kann sich Vorstellen, dass sogar bei einer kleinen Zahl von Buchstaben, wie die 57 Buchstaben von eben, sich eine recht große Menge Variationen ergibt (um genau zu sein n!/(n-k!) mit n=57 und k=2, wobei nochmal n dazukommen, weil man ja aa, bb etc. mitberücksichtigen will). Damit der Speicherbedarf auch hier nicht unnötig groß wird, habe ich für die Bigramme keine Groß-/Kleinschreibung unterschieden, sowie das Leerzeichen hinzugefügt, damit auch die Buchstaben am Wortanfang und am Wortende unterschieden werden.

Hier nun die Ergebnisse der Bigramm-Statistiken:

analyserBigrammFrequencyAnalyser
languagedeutsch
total4114752
tokenaaabbaaccaadda
abs.000.959012.770003.780013.462000.044002.390029.894
rel.0,000230,003100,000920,003270,000010,000580,00727
tokenaeeaaffaaggaah
abs.000.145001.532005.062006.119008.158005.894008.242
rel.0,000040,000370,001230,001490,001980,001430,00200
tokenhaaiiaajjaakka
abs.021.111000.478001.572000.121000.829001.304004.913
rel.0,005130,000120,000380,000030,000200,000320,00119
tokenallaammaannaao
abs.033.523008.981007.619010.836042.775009.799000.017
rel.0,008150,002180,001850,002630,010400,002380,00000
tokenoaappaaqqaarra
abs.000.039001.063001.586000.000000.000019.763017.191
rel.0,000010,000260,000390,000000,000000,004800,00418
tokenassaattaauuaav
abs.024.777008.323017.697011.399030.879000.797000.557
rel.0,006020,002020,004300,002770,007500,000190,00014
tokenvaawwaaxxaayya
abs.000.327000.029013.317000.088000.021000.064000.014
rel.0,000080,000010,003240,000020,000010,000020,00000
tokenazzaäaöa
abs.000.519000.404000.000000.001000.000000.000000.000
rel.0,000130,000100,000000,000000,000000,000000,00000
tokenüaa��aa abbbc
abs.000.000007.841000.040000.000000.000000.124000.026
rel.0,000000,001910,000010,000000,000000,000030,00001
tokencbbddbbeebbffb
abs.000.005000.030000.121048.606012.083000.062000.145
rel.0,000000,000010,000030,011810,002940,000020,00004
tokenbggbbhhbbiibbj
abs.001.017000.089000.518000.237003.423002.470000.951
rel.0,000250,000020,000130,000060,000830,000600,00023
tokenjbbkkbbllbbmmb
abs.000.000000.355000.137004.230009.766000.026000.455
rel.0,000000,000090,000030,001030,002370,000010,00011
tokenbnnbboobbppbbq
abs.000.236002.409000.592003.152000.008000.001000.001
rel.0,000060,000590,000140,000770,000000,000000,00000
tokenqbbrrbbssbbttb
abs.000.000003.635005.083008.341000.829002.428001.302
rel.0,000000,000880,001240,002030,000200,000590,00032
tokenbuubbvvbbwwbbx
abs.001.183002.768000.005000.001000.169000.000000.000
rel.0,000290,000670,000000,000000,000040,000000,00000
tokenxbbyybbzzbäb
abs.000.000000.055000.007000.341000.097000.203000.083
rel.0,000000,000010,000000,000080,000020,000050,00002
tokenöbübb��bb
abs.000.074000.042000.276005.838000.000000.104000.000
rel.0,000020,000010,000070,001420,000000,000030,00000
token bcccddcceeccf
abs.000.000000.022000.000000.203000.094005.543000.000
rel.0,000000,000010,000000,000050,000020,001350,00000
tokenfccggcchhcciic
abs.000.010000.000000.001142.026000.030000.188060.263
rel.0,000000,000000,000000,034520,000010,000050,01465
tokencjjcckkccllccm
abs.000.000000.000008.174000.034000.065004.531000.001
rel.0,000000,000000,001990,000010,000020,001100,00000
tokenmccnnccooccppc
abs.000.015000.004000.636000.191006.315000.002000.010
rel.0,000000,000000,000150,000050,001530,000000,00000
tokencqqccrrccsscct
abs.000.002000.000000.045005.718000.001032.142000.025
rel.0,000000,000000,000010,001390,000000,007810,00001
tokentccuuccvvccwwc
abs.000.075000.022008.714000.000000.000000.000000.000
rel.0,000020,000010,002120,000000,000000,000000,00000
tokencxxccyycczzc
abs.000.000000.006000.000000.226000.002000.020000.001
rel.0,000000,000000,000000,000050,000000,000000,00000
tokenäcöcücc��c
abs.001.504000.000000.756000.000003.508000.000000.053
rel.0,000370,000000,000180,000000,000850,000000,00001
tokenc cdddeeddffd
abs.000.000000.000000.215110.112012.164000.057000.023
rel.0,000000,000000,000050,026760,002960,000010,00001
tokendggddhhddiiddj
abs.000.085000.076000.147000.394042.322004.732000.013
rel.0,000020,000020,000040,000100,010290,001150,00000
tokenjddkkddllddmmd
abs.000.000000.037000.023002.331003.482000.124000.475
rel.0,000000,000010,000010,000570,000850,000030,00012
tokendnnddooddppddq
abs.000.961070.061002.617004.497000.165000.002000.003
rel.0,000230,017030,000640,001090,000040,000000,00000
tokenqddrrddssddttd
abs.000.000002.879012.783001.109000.843000.837000.230
rel.0,000000,000700,003110,000270,000200,000200,00006
tokenduuddvvddwwddx
abs.008.059000.838000.048000.000000.251000.000000.000
rel.0,001960,000200,000010,000000,000060,000000,00000
tokenxddyyddzzdäd
abs.000.000000.065000.008000.114000.062000.128000.818
rel.0,000000,000020,000000,000030,000020,000030,00020
tokenödüdd��dd
abs.000.015000.136000.417000.236000.000000.028000.000
rel.0,000000,000030,000100,000060,000000,000010,00000
token deeeffeeggeeh
abs.000.000002.177005.029009.536018.611069.917017.680
rel.0,000000,000530,001220,002320,004520,016990,00430
tokenheeiieejjeekke
abs.053.346110.609083.065000.177004.946003.457016.143
rel.0,012960,026880,020190,000040,001200,000840,00392
tokenelleemmeenneeo
abs.034.093031.000024.357020.784185.520052.316001.104
rel.0,008290,007530,005920,005050,045090,012710,00027
tokenoeeppeeqqeerre
abs.000.125000.958002.062000.174000.000167.314040.612
rel.0,000030,000230,000500,000040,000000,040660,00987
tokenesseetteeuueev
abs.065.137057.543016.990076.764007.870003.272000.451
rel.0,015830,013980,004130,018660,001910,000800,00011
tokenveewweexxeeyye
abs.011.277008.444023.461000.457000.058000.197000.111
rel.0,002740,002050,005700,000110,000010,000050,00003
tokenezzeäeöe
abs.001.863010.662000.053000.012000.020000.001000.047
rel.0,000450,002590,000010,000000,000000,000000,00001
tokenüee��ee efffg
abs.000.001001.772004.905000.000000.000006.602001.066
rel.0,000000,000430,001190,000000,000000,001600,00026
tokengffhhffiiffjjf
abs.000.455000.335000.217002.961004.410000.003000.000
rel.0,000110,000080,000050,000720,001070,000000,00000
tokenfkkffllffmmffn
abs.000.124000.153002.191000.605000.183000.159000.546
rel.0,000030,000040,000530,000150,000040,000040,00013
tokennffooffppffqqf
abs.004.274002.622002.082000.040002.450000.000000.000
rel.0,001040,000640,000510,000010,000600,000000,00000
tokenfrrffssffttffu
abs.003.269004.756000.928000.653008.510000.724001.140
rel.0,000790,001160,000230,000160,002070,000180,00028
tokenuffvvffwwffxxf
abs.012.526000.019000.001000.105000.000000.000000.001
rel.0,003040,000000,000000,000030,000000,000000,00000
tokenfyyffzzfäf
abs.000.000000.002000.350000.029001.376000.895000.127
rel.0,000000,000000,000090,000010,000330,000220,00003
tokenöfüff��ff f
abs.000.522006.339000.150000.000000.036000.000000.000
rel.0,000130,001540,000040,000000,000010,000000,00000
tokenggghhggiiggjjg
abs.000.150000.100000.488004.202021.705000.023000.000
rel.0,000040,000020,000120,001020,005270,000010,00000
tokengkkggllggmmggn
abs.002.162000.305007.258003.500000.163000.370001.534
rel.0,000530,000070,001760,000850,000040,000090,00037
tokennggooggppggqqg
abs.038.866000.499002.560000.005000.000000.001000.000
rel.0,009450,000120,000620,000000,000000,000000,00000
tokengrrggssggttggu
abs.006.191006.364003.583001.870006.446001.533004.245
rel.0,001500,001550,000870,000450,001570,000370,00103
tokenuggvvggwwggxxg
abs.004.334000.006000.019000.064000.000000.000000.000
rel.0,001050,000000,000000,000020,000000,000000,00000
tokengyyggzzgäg
abs.000.011000.008000.030000.079000.459000.637000.212
rel.0,000000,000000,000010,000020,000110,000150,00005
tokenögügg��gg g
abs.002.002000.436000.775000.000000.064000.000000.000
rel.0,000490,000110,000190,000000,000020,000000,00000
tokenhhhiihhjjhhkkh
abs.000.549011.252011.758000.045000.000002.508000.108
rel.0,000130,002730,002860,000010,000000,000610,00003
tokenhllhhmmhhnnhho
abs.009.194000.790005.819000.111006.972004.139003.473
rel.0,002230,000190,001410,000030,001690,001010,00084
tokenohhpphhqqhhrrh
abs.004.275000.015001.617000.003000.000020.640005.711
rel.0,001040,000000,000390,000000,000000,005020,00139
tokenhsshhtthhuuhhv
abs.003.383000.720028.383004.109002.906001.453000.070
rel.0,000820,000170,006900,001000,000710,000350,00002
tokenvhhwwhhxxhhyyh
abs.000.000002.644000.000000.000000.001000.427000.002
rel.0,000000,000640,000000,000000,000000,000100,00000
tokenhzzhähöh
abs.000.269000.031003.358002.338002.543000.642000.624
rel.0,000070,000010,000820,000570,000620,000160,00015
tokenühh��hh hiiij
abs.003.052000.000000.355000.000000.000000.021000.010
rel.0,000740,000000,000090,000000,000000,000010,00000
tokenjiikkiilliimmi
abs.000.003002.445000.262010.644029.165009.152014.258
rel.0,000000,000590,000060,002590,007090,002220,00347
tokeninniiooiippiiq
abs.097.166020.518004.514000.166001.147002.538000.026
rel.0,023610,004990,001100,000040,000280,000620,00001
tokenqiirriissiitti
abs.000.000013.084018.735034.455028.732037.733018.644
rel.0,000000,003180,004550,008370,006980,009170,00453
tokeniuuiivviiwwiix
abs.000.461000.395002.550003.160000.044017.838000.092
rel.0,000110,000100,000620,000770,000010,004340,00002
tokenxiiyyiizziäi
abs.000.621000.000000.014000.930004.527000.033000.046
rel.0,000150,000000,000000,000230,001100,000010,00001
tokenöiüii��ii
abs.000.119000.000000.000000.000002.319000.612000.000
rel.0,000030,000000,000000,000000,000560,000150,00000
token ijjjkkjjlljjm
abs.000.000000.000000.000000.000000.000000.015000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenmjjnnjjoojjppj
abs.000.049000.000000.055000.095000.025000.000000.000
rel.0,000010,000000,000010,000020,000010,000000,00000
tokenjqqjjrrjjssjjt
abs.000.000000.000000.000000.104000.000000.250000.000
rel.0,000000,000000,000000,000030,000000,000060,00000
tokentjjuujjvvjjwwj
abs.000.005000.386000.016000.000000.000000.000000.001
rel.0,000000,000090,000000,000000,000000,000000,00000
tokenjxxjjyyjjzzj
abs.000.000000.000000.000000.000000.000000.000000.102
rel.0,000000,000000,000000,000000,000000,000000,00002
tokenäjöjüjj��j
abs.000.000000.001000.000000.069000.000000.000000.000
rel.0,000000,000000,000000,000020,000000,000000,00000
tokenj jkkkllkkmmk
abs.000.000000.000000.147004.779001.567000.173000.259
rel.0,000000,000000,000040,001160,000380,000040,00006
tokenknnkkookkppkkq
abs.000.702005.225003.562000.849000.005000.004000.000
rel.0,000170,001270,000870,000210,000000,000000,00000
tokenqkkrrkksskkttk
abs.000.000001.778007.996001.601000.801006.241000.136
rel.0,000000,000430,001940,000390,000190,001520,00003
tokenkuukkvvkkwwkkx
abs.001.672000.733000.025000.001000.164000.000000.000
rel.0,000410,000180,000010,000000,000040,000000,00000
tokenxkkyykkzzkäk
abs.000.003000.015000.007000.228000.011000.226000.009
rel.0,000000,000000,000000,000060,000000,000050,00000
tokenökükk��kk
abs.002.122000.036000.609000.002000.000000.013000.000
rel.0,000520,000010,000150,000000,000000,000000,00000
token klllmmllnnllo
abs.000.000022.903001.327001.539002.096002.223005.118
rel.0,000000,005570,000320,000370,000510,000540,00124
tokenollppllqqllrrl
abs.011.658000.188000.402000.001000.000000.160005.459
rel.0,002830,000050,000100,000000,000000,000040,00133
tokenlssllttlluullv
abs.014.663000.564015.387004.093005.311002.638000.260
rel.0,003560,000140,003740,000990,001290,000640,00006
tokenvllwwllxxllyyl
abs.000.002000.163000.001000.000000.000000.202000.030
rel.0,000000,000040,000000,000000,000000,000050,00001
tokenlzzlälöl
abs.000.415000.632002.302002.157000.563000.507001.265
rel.0,000100,000150,000560,000520,000140,000120,00031
tokenüll��ll lmmmn
abs.000.966000.000000.371000.000000.000010.147000.263
rel.0,000230,000000,000090,000000,000000,002470,00006
tokennmmoommppmmqqm
abs.002.180001.668005.554002.441000.003000.001000.000
rel.0,000530,000410,001350,000590,000000,000000,00000
tokenmrrmmssmmttmmu
abs.000.094004.022001.175001.669002.762000.361004.052
rel.0,000020,000980,000290,000410,000670,000090,00098
tokenummvvmmwwmmxxm
abs.008.230000.018000.001000.087000.001000.000000.001
rel.0,002000,000000,000000,000020,000000,000000,00000
tokenmyymmzzmäm
abs.000.032000.111000.056000.054000.860000.982001.827
rel.0,000010,000030,000010,000010,000210,000240,00044
tokenömümm��mm m
abs.000.097001.365000.306000.000000.060000.000000.000
rel.0,000020,000330,000070,000000,000010,000000,00000
tokennnnoonnppnnqqn
abs.021.409005.607020.037000.153000.039000.015000.000
rel.0,005200,001360,004870,000040,000010,000000,00000
tokennrrnnssnnttnnu
abs.000.722010.444023.749000.128022.562001.719010.154
rel.0,000180,002540,005770,000030,005480,000420,00247
tokenunnvvnnwwnnxxn
abs.070.942000.554000.001001.520000.000000.002000.000
rel.0,017240,000130,000000,000370,000000,000000,00000
tokennyynnzznän
abs.000.019000.653007.711000.027001.577004.993000.299
rel.0,000000,000160,001870,000010,000380,001210,00007
tokenönünn��nn n
abs.003.107001.157002.309000.000000.040000.000000.000
rel.0,000760,000280,000560,000000,000010,000000,00000
tokenoooppooqqoorro
abs.000.119001.580001.409000.000000.000018.230004.763
rel.0,000030,000380,000340,000000,000000,004430,00116
tokenossoottoouuoov
abs.004.099018.474003.312001.785000.272000.037000.379
rel.0,001000,004490,000800,000430,000070,000010,00009
tokenvoowwooxxooyyo
abs.014.390000.575006.422000.019000.004000.033000.005
rel.0,003500,000140,001560,000000,000000,000010,00000
tokenozzoäoöo
abs.001.060000.621000.000000.000000.000000.000000.000
rel.0,000260,000150,000000,000000,000000,000000,00000
tokenüoo��oo opppq
abs.000.000002.687000.007000.000000.000000.915000.000
rel.0,000000,000650,000000,000000,000000,000220,00000
tokenqpprrppssppttp
abs.000.000004.586000.552000.225004.712001.884000.103
rel.0,000000,001110,000130,000050,001150,000460,00003
tokenpuuppvvppwwppx
abs.000.578001.709000.001000.000000.000000.000000.000
rel.0,000140,000420,000000,000000,000000,000000,00000
tokenxppyyppzzpäp
abs.000.080000.002000.191000.302000.019000.479000.006
rel.0,000020,000000,000050,000070,000000,000120,00000
tokenöpüpp��pp
abs.000.051000.269000.053000.266000.000000.013000.000
rel.0,000010,000070,000010,000060,000000,000000,00000
token pqqqrrqqssqqt
abs.000.000000.000000.000000.039000.000000.028000.000
rel.0,000000,000000,000000,000010,000000,000010,00000
tokentqquuqqvvqqwwq
abs.000.018000.461000.001000.000000.000000.000000.000
rel.0,000000,000110,000000,000000,000000,000000,00000
tokenqxxqqyyqqzzq
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenäqöqüqq��q
abs.000.014000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenq qrrrssrrttr
abs.000.000000.000003.175018.467000.421016.440009.156
rel.0,000000,000000,000770,004490,000100,004000,00223
tokenruurrvvrrwwrrx
abs.009.122017.303001.121000.009002.772000.001000.110
rel.0,002220,004210,000270,000000,000670,000000,00003
tokenxrryyrrzzrär
abs.000.000000.008000.161003.621000.018002.774003.101
rel.0,000000,000000,000040,000880,000000,000670,00075
tokenörürr��rr
abs.001.063001.998004.265007.682000.000000.067000.000
rel.0,000260,000490,001040,001870,000000,000020,00000
token rsssttssuussv
abs.000.000012.973053.791010.925002.914014.621000.499
rel.0,000000,003150,013070,002660,000710,003550,00012
tokenvsswwssxxssyys
abs.000.003001.167000.030000.000000.007000.733000.862
rel.0,000000,000280,000010,000000,000000,000180,00021
tokenszzsäsös
abs.001.228000.148000.622000.555000.294000.984000.178
rel.0,000300,000040,000150,000130,000070,000240,00004
tokenüss��ss stttu
abs.001.209000.000000.099000.000000.000011.337006.623
rel.0,000290,000000,000020,000000,000000,002760,00161
tokenuttvvttwwttxxt
abs.007.820000.235000.006003.570000.015000.000000.171
rel.0,001900,000060,000000,000870,000000,000000,00004
tokentyyttzztät
abs.000.033000.175009.819003.505004.362004.117000.260
rel.0,000010,000040,002390,000850,001060,001000,00006
tokenötütt��tt t
abs.000.669001.446001.067000.000005.454000.000000.000
rel.0,000160,000350,000260,000000,001330,000000,00000
tokenuuuvvuuwwuuxxu
abs.001.071000.205000.017000.235005.037000.027000.014
rel.0,000260,000050,000000,000060,001220,000010,00000
tokenuyyuuzzuäu
abs.000.002000.002000.221022.661000.047001.746000.000
rel.0,000000,000000,000050,005510,000010,000420,00000
tokenöuüuu��uu u
abs.000.000000.003000.000007.210000.096000.000000.000
rel.0,000000,000000,000000,001750,000020,000000,00000
tokenvvvwwvvxxvvyyv
abs.000.000000.000000.000000.000000.001000.002000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenvzzvävöv
abs.000.000000.020000.018000.000000.275000.006000.000
rel.0,000000,000000,000000,000000,000070,000000,00000
tokenüvv��vv vwwwx
abs.000.000000.000000.072000.000000.000000.001000.000
rel.0,000000,000000,000020,000000,000000,000000,00000
tokenxwwyywwzzwäw
abs.000.001000.001000.000000.000002.724002.223000.000
rel.0,000000,000000,000000,000000,000660,000540,00000
tokenöwüww��ww
abs.000.282000.023001.663000.000000.000000.010000.000
rel.0,000070,000010,000400,000000,000000,000000,00000
token wxxxyyxxzzx
abs.000.000000.000000.001000.001000.028000.000000.001
rel.0,000000,000000,000000,000000,000010,000000,00000
tokenäxöxüxx��x
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenx xyyyzzyäy
abs.000.000000.000000.000000.002000.003000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenöyüyy��yy
abs.000.000000.000000.000000.000000.003000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token yzzäzöz
abs.000.000000.039000.408000.009000.152000.001000.268
rel.0,000000,000010,000100,000000,000040,000000,00007
tokenüzz��zz zäääö
abs.000.000000.000000.012000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenöääüüää��ää ä
abs.000.000000.000000.000000.870000.000000.000000.000
rel.0,000000,000000,000000,000210,000000,000000,00000
tokenöööüüöö��öö ö
abs.000.000000.000000.000000.682000.000000.000000.000
rel.0,000000,000000,000000,000170,000000,000000,00000
tokenüüü��üü ü���
abs.000.000000.594000.000000.000000.000000.000000.000
rel.0,000000,000140,000000,000000,000000,000000,00000
token �
abs.000.000000.000
rel.0,000000,00000

analyserBigrammFrequencyAnalyser
languageenglish
total3854701
tokenaaabbaaccaadda
abs.000.005008.608004.668014.817014.406021.033005.373
rel.0,000000,002230,001210,003840,003740,005460,00139
tokenaeeaaffaaggaah
abs.000.628028.550003.304007.334006.813005.336000.356
rel.0,000160,007410,000860,001900,001770,001380,00009
tokenhaaiiaajjaakka
abs.049.670016.605005.315000.136000.226005.583000.268
rel.0,012890,004310,001380,000040,000060,001450,00007
tokenallaammaannaao
abs.027.313016.001012.182019.433082.334006.554000.101
rel.0,007090,004150,003160,005040,021360,001700,00003
tokenoaappaaqqaarra
abs.002.460006.465009.410000.017000.000037.814018.218
rel.0,000640,001680,002440,000000,000000,009810,00473
tokenassaattaauuaav
abs.040.436012.015050.467015.157004.199003.195011.087
rel.0,010490,003120,013090,003930,001090,000830,00288
tokenvaawwaaxxaayya
abs.003.445003.681025.698000.257000.954011.113000.464
rel.0,000890,000950,006670,000070,000250,002880,00012
tokenazzaäaöa
abs.000.422000.842000.000000.000000.000000.000000.000
rel.0,000110,000220,000000,000000,000000,000000,00000
tokenüaa��aa abbbc
abs.000.000000.000000.000000.000000.000000.498000.006
rel.0,000000,000000,000000,000000,000000,000130,00000
tokencbbddbbeebbffb
abs.000.001000.041000.026023.786000.674000.001000.002
rel.0,000000,000010,000010,006170,000170,000000,00000
tokenbggbbhhbbiibbj
abs.000.002000.093000.016000.189002.668002.297000.699
rel.0,000000,000020,000000,000050,000690,000600,00018
tokenjbbkkbbllbbmmb
abs.000.000000.000000.013008.941000.184000.077002.637
rel.0,000000,000000,000000,002320,000050,000020,00068
tokenbnnbboobbppbbq
abs.000.021000.127007.713002.651000.000000.019000.000
rel.0,000010,000030,002000,000690,000000,000000,00000
tokenqbbrrbbssbbttb
abs.000.000004.037000.701001.295000.468000.763000.039
rel.0,000000,001050,000180,000340,000120,000200,00001
tokenbuubbvvbbwwbbx
abs.008.247002.682000.077000.000000.005000.029000.000
rel.0,002140,000700,000020,000000,000000,000010,00000
tokenxbbyybbzzbäb
abs.000.000005.488000.418000.000000.001000.000000.000
rel.0,000000,001420,000110,000000,000000,000000,00000
tokenöbübb��bb
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token bcccddcceeccf
abs.000.000002.584000.014000.034021.156012.049000.000
rel.0,000000,000670,000000,000010,005490,003130,00000
tokenfccggcchhcciic
abs.000.001000.000000.000020.009000.009005.852017.345
rel.0,000000,000000,000000,005190,000000,001520,00450
tokencjjcckkccllccm
abs.000.000000.000007.549000.006004.270000.295000.000
rel.0,000000,000000,001960,000000,001110,000080,00000
tokenmccnnccooccppc
abs.000.020000.012012.227022.943003.712000.000000.004
rel.0,000010,000000,003170,005950,000960,000000,00000
tokencqqccrrccsscct
abs.000.318000.000004.525003.404000.187003.593009.788
rel.0,000080,000000,001170,000880,000050,000930,00254
tokentccuuccvvccwwc
abs.001.546003.984006.084000.000000.000000.028000.121
rel.0,000400,001030,001580,000000,000000,000010,00003
tokencxxccyycczzc
abs.000.000001.257001.263000.061000.001000.005000.000
rel.0,000000,000330,000330,000020,000000,000000,00000
tokenäcöcücc��c
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenc cdddeeddffd
abs.000.000000.000001.971025.083053.677000.147000.001
rel.0,000000,000000,000510,006510,013930,000040,00000
tokendggddhhddiiddj
abs.001.027000.064000.068000.083014.223013.505000.092
rel.0,000270,000020,000020,000020,003690,003500,00002
tokenjddkkddllddmmd
abs.000.000000.038000.002001.774014.067000.583000.001
rel.0,000000,000010,000000,000460,003650,000150,00000
tokendnnddooddppddq
abs.001.736062.507009.862006.096000.021000.004000.066
rel.0,000450,016220,002560,001580,000010,000000,00002
tokenqddrrddssddttd
abs.000.000003.802007.297004.908000.127000.057000.014
rel.0,000000,000990,001890,001270,000030,000010,00000
tokenduuddvvddwwddx
abs.002.834002.500000.855000.000000.196000.315000.000
rel.0,000740,000650,000220,000000,000050,000080,00000
tokenxddyyddzzdäd
abs.000.000002.498000.251000.002000.000000.000000.000
rel.0,000000,000650,000070,000000,000000,000000,00000
tokenödüdd��dd
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token deeeffeeggeeh
abs.000.000018.107005.715009.124002.785012.560000.884
rel.0,000000,004700,001480,002370,000720,003260,00023
tokenheeiieejjeekke
abs.141.242006.946012.500000.144001.251000.767012.524
rel.0,036640,001800,003240,000040,000320,000200,00325
tokenelleemmeenneeo
abs.020.230032.648012.519033.276053.617029.781001.646
rel.0,005250,008470,003250,008630,013910,007730,00043
tokenoeeppeeqqeerre
abs.001.126005.691015.289001.186000.000086.349066.975
rel.0,000290,001480,003970,000310,000000,022400,01737
tokenesseetteeuueev
abs.039.878033.051017.720039.233000.427003.917010.010
rel.0,010350,008570,004600,010180,000110,001020,00260
tokenveewweexxeeyye
abs.035.091004.245017.181006.629000.623009.139004.267
rel.0,009100,001100,004460,001720,000160,002370,00111
tokenezzeäeöe
abs.000.124001.113000.000000.000000.000000.000000.000
rel.0,000030,000290,000000,000000,000000,000000,00000
tokenüee��ee efffg
abs.000.000000.000000.000000.000000.000005.003000.002
rel.0,000000,000000,000000,000000,000000,001300,00000
tokengffhhffiiffjjf
abs.000.009000.001000.110008.679007.551000.000000.000
rel.0,000000,000000,000030,002250,001960,000000,00000
tokenfkkffllffmmffn
abs.000.000000.167002.491003.403000.001000.254000.020
rel.0,000000,000040,000650,000880,000000,000070,00001
tokennffooffppffqqf
abs.001.670018.309040.619000.003000.032000.000000.000
rel.0,000430,004750,010540,000000,000010,000000,00000
tokenfrrffssffttffu
abs.007.677001.288000.104000.519004.539000.321003.694
rel.0,001990,000330,000030,000130,001180,000080,00096
tokenuffvvffwwffxxf
abs.001.077000.000000.000000.015000.105000.000000.007
rel.0,000280,000000,000000,000000,000030,000000,00000
tokenfyyffzzfäf
abs.000.205000.055000.000000.000000.000000.000000.000
rel.0,000050,000010,000000,000000,000000,000000,00000
tokenöfüff��ff f
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenggghhggiiggjjg
abs.001.209013.561000.001004.756010.797000.000000.000
rel.0,000310,003520,000000,001230,002800,000000,00000
tokengkkggllggmmggn
abs.000.000000.024002.776000.149000.099000.000001.531
rel.0,000000,000010,000720,000040,000030,000000,00040
tokennggooggppggqqg
abs.042.581006.762001.369000.000000.000000.000000.000
rel.0,011050,001750,000360,000000,000000,000000,00000
tokengrrggssggttggu
abs.006.003002.520002.175000.179000.951000.024002.488
rel.0,001560,000650,000560,000050,000250,000010,00065
tokenuggvvggwwggxxg
abs.006.674000.002000.000000.092000.004000.000000.000
rel.0,001730,000000,000000,000020,000000,000000,00000
tokengyyggzzgäg
abs.000.349000.026000.001000.001000.000000.000000.000
rel.0,000090,000010,000000,000000,000000,000000,00000
tokenögügg��gg g
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenhhhiihhjjhhkkh
abs.000.015043.257000.024000.000000.000000.016000.210
rel.0,000000,011220,000010,000000,000000,000000,00005
tokenhllhhmmhhnnhho
abs.000.344000.025000.506000.012000.277000.378020.781
rel.0,000090,000010,000130,000000,000070,000100,00539
tokenohhpphhqqhhrrh
abs.000.328000.007001.312000.054000.000002.907000.658
rel.0,000090,000000,000340,000010,000000,000750,00017
tokenhsshhtthhuuhhv
abs.000.473017.342008.812135.755003.352000.027000.001
rel.0,000120,004500,002290,035220,000870,000010,00000
tokenvhhwwhhxxhhyyh
abs.000.000000.088018.831000.000000.157001.335000.083
rel.0,000000,000020,004890,000000,000040,000350,00002
tokenhzzhähöh
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenühh��hh hiiij
abs.000.000000.000000.000000.000000.000000.014000.006
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenjiikkiilliimmi
abs.000.058002.313004.492018.179021.349015.251011.498
rel.0,000020,000600,001170,004720,005540,003960,00298
tokeninniiooiippiiq
abs.090.027009.427016.521003.687002.588004.792000.117
rel.0,023360,002450,004290,000960,000670,001240,00003
tokenqiirriissiitti
abs.000.000013.290023.327041.679015.812041.908028.643
rel.0,000000,003450,006050,010810,004100,010870,00743
tokeniuuiivviiwwiix
abs.000.579003.660007.557006.807000.007018.386000.781
rel.0,000150,000950,001960,001770,000000,004770,00020
tokenxiiyyiizziäi
abs.000.638000.000001.457001.702000.175000.000000.000
rel.0,000170,000000,000380,000440,000050,000000,00000
tokenöiüii��ii
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token ijjjkkjjlljjm
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenmjjnnjjoojjppj
abs.000.000000.000000.497000.918000.092000.000000.001
rel.0,000000,000000,000130,000240,000020,000000,00000
tokenjqqjjrrjjssjjt
abs.000.000000.000000.000000.008000.000000.005000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokentjjuujjvvjjwwj
abs.000.000001.837000.001000.000000.000000.000000.000
rel.0,000000,000480,000000,000000,000000,000000,00000
tokenjxxjjyyjjzzj
abs.000.000000.000000.001000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenäjöjüjj��j
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenj jkkkllkkmmk
abs.000.000000.000000.005000.732001.435000.279000.000
rel.0,000000,000000,000000,000190,000370,000070,00000
tokenknnkkookkppkkq
abs.003.590003.697000.266004.773000.011000.014000.000
rel.0,000930,000960,000070,001240,000000,000000,00000
tokenqkkrrkksskkttk
abs.000.000000.023003.446002.467002.203000.017000.004
rel.0,000000,000010,000890,000640,000570,000000,00000
tokenkuukkvvkkwwkkx
abs.000.268000.206000.000000.000000.087000.095000.000
rel.0,000070,000050,000000,000000,000020,000020,00000
tokenxkkyykkzzkäk
abs.000.000000.298000.004000.000000.000000.000000.000
rel.0,000000,000080,000000,000000,000000,000000,00000
tokenökükk��kk
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token klllmmllnnllo
abs.000.000027.601001.401000.224000.117002.896015.362
rel.0,000000,007160,000360,000060,000030,000750,00399
tokenollppllqqllrrl
abs.011.614000.821007.947000.002000.000000.336002.982
rel.0,003010,000210,002060,000000,000000,000090,00077
tokenlssllttlluullv
abs.003.362002.535004.657005.677002.975014.695001.039
rel.0,000870,000660,001210,001470,000770,003810,00027
tokenvllwwllxxllyyl
abs.000.001000.775000.675000.000000.000016.263000.583
rel.0,000000,000200,000180,000000,000000,004220,00015
tokenlzzlälöl
abs.000.001000.108000.000000.000000.000000.000000.000
rel.0,000000,000030,000000,000000,000000,000000,00000
tokenüll��ll lmmmn
abs.000.000000.000000.000000.000000.000002.347000.357
rel.0,000000,000000,000000,000000,000000,000610,00009
tokennmmoommppmmqqm
abs.000.473013.555024.297006.602000.042000.000000.000
rel.0,000120,003520,006300,001710,000010,000000,00000
tokenmrrmmssmmttmmu
abs.000.020004.361002.868002.016000.036000.540004.211
rel.0,000010,001130,000740,000520,000010,000140,00109
tokenummvvmmwwmmxxm
abs.003.731000.042000.012000.003000.010000.000000.000
rel.0,000970,000010,000000,000000,000000,000000,00000
tokenmyymmzzmäm
abs.005.734000.438000.000000.000000.000000.000000.000
rel.0,001490,000110,000000,000000,000000,000000,00000
tokenömümm��mm m
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokennnnoonnppnnqqn
abs.003.026021.039057.375000.170000.026000.402000.000
rel.0,000790,005460,014880,000040,000010,000100,00000
tokennrrnnssnnttnnu
abs.000.253006.600013.187000.862032.332000.544002.176
rel.0,000070,001710,003420,000220,008390,000140,00056
tokenunnvvnnwwnnxxn
abs.015.954001.682000.000000.277004.653000.214000.000
rel.0,004140,000440,000000,000070,001210,000060,00000
tokennyynnzznän
abs.005.009000.210000.039000.001000.000000.000000.000
rel.0,001300,000050,000010,000000,000000,000000,00000
tokenönünn��nn n
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenoooppooqqoorro
abs.013.746006.833010.789000.064000.000040.158023.420
rel.0,003570,001770,002800,000020,000000,010420,00608
tokenossoottoouuoov
abs.010.360015.849018.728048.952053.493000.248006.321
rel.0,002690,004110,004860,012700,013880,000060,00164
tokenvoowwooxxooyyo
abs.002.012017.670011.166000.326000.030001.583013.681
rel.0,000520,004580,002900,000080,000010,000410,00355
tokenozzoäoöo
abs.000.125000.055000.000000.000000.000000.000000.000
rel.0,000030,000010,000000,000000,000000,000000,00000
tokenüoo��oo opppq
abs.000.000000.000000.000000.000000.000005.016000.000
rel.0,000000,000000,000000,000000,000000,001300,00000
tokenqpprrppssppttp
abs.000.000011.498001.156001.834006.326003.012000.032
rel.0,000000,002980,000300,000480,001640,000780,00001
tokenpuuppvvppwwppx
abs.003.144006.941000.000000.000000.045000.051000.000
rel.0,000820,001800,000000,000000,000010,000010,00000
tokenxppyyppzzpäp
abs.001.913000.561000.328000.001000.000000.000000.000
rel.0,000500,000150,000090,000000,000000,000000,00000
tokenöpüpp��pp
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token pqqqrrqqssqqt
abs.000.000000.000000.000000.012000.000000.281000.000
rel.0,000000,000000,000000,000000,000000,000070,00000
tokentqquuqqvvqqwwq
abs.000.000004.919000.009000.000000.000000.000000.000
rel.0,000000,001280,000000,000000,000000,000000,00000
tokenqxxqqyyqqzzq
abs.000.000000.018000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenäqöqüqq��q
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenq qrrrssrrttr
abs.000.000000.000005.487015.253000.045011.535012.526
rel.0,000000,000000,001420,003960,000010,002990,00325
tokenruurrvvrrwwrrx
abs.004.137020.440001.743000.021000.572001.255000.000
rel.0,001070,005300,000450,000010,000150,000330,00000
tokenxrryyrrzzrär
abs.000.000010.101000.164000.000000.000000.000000.000
rel.0,000000,002620,000040,000000,000000,000000,00000
tokenörürr��rr
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token rsssttssuussv
abs.000.000014.489035.201008.365009.926016.981000.025
rel.0,000000,003760,009130,002170,002580,004410,00001
tokenvsswwssxxssyys
abs.000.002001.785001.118000.000000.000000.901003.507
rel.0,000000,000460,000290,000000,000000,000230,00091
tokenszzsäsös
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenüss��ss stttu
abs.000.000000.000000.000000.000000.000009.575006.789
rel.0,000000,000000,000000,000000,000000,002480,00176
tokenuttvvttwwttxxt
abs.020.904000.004000.000002.763000.141000.000001.828
rel.0,005420,000000,000000,000720,000040,000000,00047
tokentyyttzztät
abs.006.229000.881000.052000.004000.000000.000000.000
rel.0,001620,000230,000010,000000,000000,000000,00000
tokenötütt��tt t
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenuuuvvuuwwuuxxu
abs.000.007000.072000.056000.000000.099000.088000.084
rel.0,000000,000020,000010,000000,000030,000020,00002
tokenuyyuuzzuäu
abs.000.393000.013000.176000.010000.000000.000000.000
rel.0,000100,000000,000050,000000,000000,000000,00000
tokenöuüuu��uu u
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenvvvwwvvxxvvyyv
abs.000.000000.000000.001000.000000.001000.253000.000
rel.0,000000,000000,000000,000000,000000,000070,00000
tokenvzzvävöv
abs.000.000000.007000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenüvv��vv vwwwx
abs.000.000000.000000.000000.000000.000000.008000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenxwwyywwzzwäw
abs.000.001000.195000.161000.001000.036000.000000.000
rel.0,000000,000050,000040,000000,000010,000000,00000
tokenöwüww��ww
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token wxxxyyxxzzx
abs.000.000000.001000.023000.002000.000000.000000.000
rel.0,000000,000000,000010,000000,000000,000000,00000
tokenäxöxüxx��x
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenx xyyyzzyäy
abs.000.000000.000000.000000.011000.159000.000000.000
rel.0,000000,000000,000000,000000,000040,000000,00000
tokenöyüyy��yy
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token yzzäzöz
abs.000.000000.274000.000000.000000.000000.000000.000
rel.0,000000,000070,000000,000000,000000,000000,00000
tokenüzz��zz zäääö
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenöääüüää��ää ä
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenöööüüöö��öö ö
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
tokenüüü��üü ü���
abs.000.000000.000000.000000.000000.000000.000000.000
rel.0,000000,000000,000000,000000,000000,000000,00000
token �
abs.000.000000.000
rel.0,000000,00000

Ich hatte ja gesagt, dass das ganz schön viele sind. Die Betrachtung von Tri- und Nochmehr-Grammen erspare ich mir vorerst.

Häufigste kurze Schlüsselwörter

Zusätzlich habe ich eine Liste von prägnanten Schlüsselwörtern erstellt. Dies setzt natürlich bereits Vorkenntnisse einer Sprache voraus, kann aber ebenfalls automatisiert werden (was ich irgendwann vielleicht auch mal machen werde).

Meine Liste für Deutsch/Englisch besteht aus folgenden Wörtern:
ANALYSED_WORDS = “der die das the wer who wie how was what er he sie she es it wenn if dann then ein a ist is wo and und nach auf on ab from aus off mit with sein be his ich me mein my dein your”

Die Ergebnisse für diese Schlüsselwörter sehen wie folgt aus:

analyserKeywordFrequencyAnalyser
languagedeutsch
total448823
tokenderdiedasthewerwhowie
abs.030.117028.540014.341000.001000.243000.000004.048
rel.0,067100,063590,031950,000000,000540,000000,00902
tokenhowwaswhaterhesieshe
abs.000.000003.269000.000007.639000.000009.892000.000
rel.0,000000,007280,000000,017020,000000,022040,00000
tokenesitwennifdannthenein
abs.008.758000.000002.787000.000000.951000.000006.463
rel.0,019510,000000,006210,000000,002120,000000,01440
tokenaistiswoandundnach
abs.000.990011.990000.013000.675000.000028.564003.211
rel.0,002210,026710,000030,001500,000000,063640,00715
tokenaufonabfromausoffmit
abs.006.066000.003000.323000.000003.031000.000006.021
rel.0,013520,000010,000720,000000,006750,000000,01342
tokenwithseinbehisichmemein
abs.000.000004.055000.002000.000008.004000.004000.525
rel.0,000000,009030,000000,000000,017830,000010,00117
tokenmydeinyour
abs.000.000000.080000.000
rel.0,000000,000180,00000

analyserKeywordFrequencyAnalyser
languageenglish
total770195
tokenderdiedasthewerwhowie
abs.000.000000.131000.000068.566000.000002.670000.000
rel.0,000000,000170,000000,089020,000000,003470,00000
tokenhowwaswhaterhesieshe
abs.001.288015.397003.129000.109015.475000.000006.224
rel.0,001670,019990,004060,000140,020090,000000,00808
tokenesitwennifdannthenein
abs.000.001014.913000.000003.485000.000002.602000.000
rel.0,000000,019360,000000,004520,000000,003380,00000
tokenaistiswoandundnach
abs.026.793000.000008.327000.000039.049000.001000.000
rel.0,034790,000000,010810,000000,050700,000000,00000
tokenaufonabfromausoffmit
abs.000.000006.828000.003004.892000.000000.901000.001
rel.0,000000,008870,000000,006350,000000,001170,00000
tokenwithseinbehisichmemein
abs.009.293000.000007.063012.168000.000004.061000.002
rel.0,012070,000000,009170,015800,000000,005270,00000
tokenmydeinyour
abs.004.776000.000002.500
rel.0,006200,000000,00325

Die Häufigkeiten sind relativ zur Summe aller aufgetauchten Schlüsselwörter bestimmt und nicht zur Summe aller verschiedener Wörter. Dies würde ja voraussetzen, dass man alle Wörter nachhält, was wieder recht Speicherintensiv wäre.

Analyse

Nun wende ich die gewonnenen Daten auf neue Texte (hauptsächlich kürzere Nachrichtentexte aus dem Netz) an und bestimme den euklidschen Abstand zwischen den Statistiken des neuen Textes zu den der vorhandenen Sprachen. Man bedenke, dass eine kleinere Zahl einen kleineren “Abstand” zu dieser Sprache bedeutet und daher die Sprache identifiziert.

Deutscher Text “Die Mathe-Kompetenz steigt, Bericht: Neue PISA-Studie zeigt viel Stillstand – 60 Prozent der Schüler in Mathematik verbessert”

LetterFrequencyAnalyserdeutsch 0.030792362448181926
LetterFrequencyAnalyserenglish 0.08809542306684467
BigrammFrequencyAnalyserdeutsch 0.03541356776787419
BigrammFrequencyAnalyserenglish 0.09038405953807747
KeywordFrequencyAnalyserdeutsch 0.07731872047723566
KeywordFrequencyAnalyserenglish 0.20037680477688585

Deutscher Text “FUSSBALL-LEGENDE Ferenc Puskás ist tot”

LetterFrequencyAnalyserdeutsch 0.04840938315327702
LetterFrequencyAnalyserenglish 0.09643409539741049
BigrammFrequencyAnalyserdeutsch 0.043410159788185125
BigrammFrequencyAnalyserenglish 0.08768861564376292
KeywordFrequencyAnalyserdeutsch 0.08264984837990634
KeywordFrequencyAnalyserenglish 0.17524762979070838

Deutscher Text “ASYLPOLITIK – Innenminister einigen sich beim Bleiberecht”

LetterFrequencyAnalyserdeutsch 0.028461735181487875
LetterFrequencyAnalyserenglish0.10139339501365155
BigrammFrequencyAnalyserdeutsch 0.035081022629477016
BigrammFrequencyAnalyserenglish 0.09442753487921235
KeywordFrequencyAnalyserdeutsch 0.07388615347744187
KeywordFrequencyAnalyserenglish 0.18176011202277478

Englischer Text “Javadoc InetAddress”

LetterFrequencyAnalyserdeutsch 0.09458231868141383
LetterFrequencyAnalyserenglish0.061178755245269195
BigrammFrequencyAnalyserdeutsch 0.09549646308136638
BigrammFrequencyAnalyserenglish 0.06517625457959006
KeywordFrequencyAnalyserdeutsch 0.19150286462462796
KeywordFrequencyAnalyserenglish 0.07513373491102585

Englischer Text: “US Senate okays nuclear technology transfer to India”

LetterFrequencyAnalyserdeutsch 0.09298511340726828
LetterFrequencyAnalyserenglish0.038985710306149904
BigrammFrequencyAnalyserdeutsch 0.09189667820735391
BigrammFrequencyAnalyserenglish 0.04695693869485645
KeywordFrequencyAnalyserdeutsch 0.240164026452281
KeywordFrequencyAnalyserenglish 0.11098611557387729

Fazit

Allein schon die Unigramm-Analyse liefert bei relativ kurzen Texten in “klarer” Sprache gute Ergebnisse. Zu testen ist, wie die Analyse auf “denglische” Texte und auf Fachtexte mit vielen englischen Vokabeln reagiert.

Selbst die Keywordanalyse bietet bereits gute Resultate und ist sehr Speicher- und Performanzschonend. Dies liegt aber wohl auch daran, dass ich explizit Wörter ausgewählt habe, die kurz und in beiden Sprachen disjunkt vorkommen. Eine automatische Extraktion von solchen Schlüsselwörtern wäre wohl weit aus Speicher- und Performanzfressender, zumal jedes Wort, dass vorkommt, abgelegt werden müsste. Nach meiner Erfahrung in einem anderen Projekt von mir können das pro Sprache schon mal einige hunderttausend Wörter werden. Hier sollte sicherlich eine sinnvolle Beschränkung der Schlüsselwörter aushelfen, wie z. B. das ein Schlüsselwort maximal 5 Zeichen haben darf. Ich werde dem zu einem späteren Zeitpunkt nachgehen.

Als nächsten Schritt werde ich, wie in der Arbeit beschrieben, die durchschnittliche Wortlänge ermitteln. Und natürlich steht ein rieser Berg an Refactoring vor, damit die API etwas sauberer und flexibler wird.

So long…