{"id":234,"date":"2015-03-23T17:59:44","date_gmt":"2015-03-23T16:59:44","guid":{"rendered":"http:\/\/www.mlguru.cz\/?p=234"},"modified":"2017-08-12T15:47:01","modified_gmt":"2017-08-12T13:47:01","slug":"word2vec-jednoducha-aritmetika-se-slovy","status":"publish","type":"post","link":"https:\/\/www.mlguru.com\/cs\/word2vec-jednoducha-aritmetika-se-slovy\/","title":{"rendered":"word2vec \u2013 jednoduch\u00e1 aritmetika se slovy"},"content":{"rendered":"<p>Kdy\u017e psal Tom\u00e1\u0161 Mikolov svoji <a href=\"http:\/\/www.fit.vutbr.cz\/~imikolov\/rnnlm\/thesis.pdf\">diserta\u010dn\u00ed pr\u00e1ci<\/a>, jist\u011b netu\u0161il, jak\u00fd rozruch jej\u00ed d\u016fsledky v celosv\u011btov\u00e9 NLP komunit\u011b vyvolaj\u00ed. Pr\u00e1ce se zab\u00fdvala jazykov\u00fdmi modely postaven\u00fdmi na neuronov\u00fdch s\u00edt\u00edch. Jazykov\u00e9 modely, kter\u00e9 se b\u011b\u017en\u011b vyu\u017e\u00edvaj\u00ed p\u0159i strojov\u00e9m zpracov\u00e1n\u00ed \u0159e\u010di nebo automatick\u00e9m p\u0159ekladu, jsou matematick\u00e9 modely, umo\u017e\u0148uj\u00edc\u00ed predikovat slova n\u00e1sleduj\u00edc\u00ed po n\u011bjak\u00e9 posloupnosti slov tak, aby vznikl\u00fd text co nejl\u00e9pe zapadal do dan\u00e9ho jazyka. Pokud tedy nap\u0159. m\u00e1me posloupnost slov \u201e\u0160el do lesa na\u201c, je v \u010de\u0161tin\u011b mnohem pravd\u011bpodobn\u011bj\u0161\u00ed, \u017ee bude n\u00e1sledovat \u201ehouby\u201c nebo \u201ed\u0159evo\u201c ne\u017e \u201emikroskop\u201c nebo \u201ev\u00edtr\u201c. Neuronov\u00e1 s\u00ed\u0165 tedy \u0159e\u0161\u00ed relativn\u011b jednoduchou \u00falohu strojov\u00e9ho u\u010den\u00ed s u\u010ditelem, kde jako tr\u00e9novac\u00ed data slou\u017e\u00ed velk\u00fd korpus text\u016f. Mikolov si v\u0161ak v\u0161iml, \u017ee synaptick\u00e9 v\u00e1hy neuronov\u00e9 s\u00edt\u011b mezi vstupem a skrytou vrstvou s\u00edt\u011b p\u0159edstavuj\u00ed vektorovou reprezentaci slov v n-rozm\u011brn\u00e9m prostoru, maj\u00edc\u00ed velmi zaj\u00edmav\u00e9 vlastnosti.<\/p>\n<p>Prvn\u00edm d\u016fle\u017eit\u00fdm rysem je, \u017ee slova, kter\u00e1 jsou si v\u00fdznamov\u011b podobn\u00e1, le\u017e\u00ed v dan\u00e9m prostoru bl\u00edzko sebe a naopak slova nepodobn\u00e1 maj\u00ed od sebe daleko. Mnohem zaj\u00edmav\u011bj\u0161\u00ed je ale vlastnost, kterou Mikolov naz\u00fdv\u00e1 \u201elinguistic regularity\u201c. Rozum\u00ed se t\u00edm zejm\u00e9na to, \u017ee velkou \u010d\u00e1st s\u00e9mantick\u00fdch i morfologick\u00fdch vlastnost\u00ed slov je mo\u017en\u00e9 reprezentovat unik\u00e1tn\u00edm vektorem. Pokud tento vektor p\u0159i\u010dteme k vektoru libovoln\u00e9ho slova, posuneme se k t\u011bsn\u00e9 bl\u00edzkosti vektoru slova, kter\u00e9 se od p\u016fvodn\u00edho li\u0161\u00ed pr\u00e1v\u011b touto vlastnost\u00ed (pokud takov\u00e9 slovo existuje). Jednoduch\u00e1 ilustrace ve dvourozm\u011brn\u00e9m prostoru je zn\u00e1zorn\u011bna na obr\u00e1zku 1 vlevo.<\/p>\n<div id=\"attachment_236\" style=\"width: 691px\" class=\"wp-caption aligncenter\"><a href=\"http:\/\/www.mlguru.cz\/wp-content\/uploads\/2015\/03\/w2v.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-236\" class=\"wp-image-236\" src=\"http:\/\/www.mlguru.cz\/wp-content\/uploads\/2015\/03\/w2v.png\" alt=\"word2vec\" width=\"681\" height=\"244\" srcset=\"https:\/\/www.mlguru.com\/wp-content\/uploads\/2015\/03\/w2v.png 1005w, https:\/\/www.mlguru.com\/wp-content\/uploads\/2015\/03\/w2v-300x107.png 300w\" sizes=\"auto, (max-width: 681px) 100vw, 681px\" \/><\/a><p id=\"caption-attachment-236\" class=\"wp-caption-text\">Obr\u00e1zek 1: Vektorov\u00e1 reprezentace slov. Zdroj: T. Mikolovov et al. :\u00a0Linguistic Regularities in Continuous Space Word Representations, NAACL 2013.<\/p><\/div>\n<p>Modr\u00e9 \u0161ipky zde p\u0159edstavuj\u00ed vektory, reprezentuj\u00edc\u00ed vlastnost zm\u011bny\u00a0pohlav\u00ed z\u00a0mu\u017esk\u00e9ho na \u017eensk\u00e9. Z\u00a0obr\u00e1zku je vid\u011bt, \u017ee se v\u017edy jedn\u00e1 o vektor stejn\u00e9ho sm\u011bru a velikosti. Lze ho tedy jednodu\u0161e spo\u010d\u00edtat ode\u010dten\u00edm vektorov\u00e9 reprezentace libovoln\u00e9 dvojice slov, reprezentuj\u00edc\u00edch objekty li\u0161\u00edc\u00ed se pouze pohlav\u00edm. Ozna\u010dme vektorovou reprezentaci slova s jako<em> vec<\/em>(s)<em>. <\/em>Vektor zm\u011bny mu\u017esk\u00e9ho pohlav\u00ed na \u017eensk\u00e9 potom z\u00edsk\u00e1me nap\u0159\u00edklad jako <em>vec<\/em>(\"woman\") \u2013 <em>vec<\/em>(\"man\") nebo <em>vec<\/em>(\"queen\") \u2013 <em>vec<\/em>(\"king\")<em>. <\/em>Vektor zm\u011bny \u017eensk\u00e9ho pohlav\u00ed na mu\u017esk\u00e9 bude m\u00edt stejnou velikost jen opa\u010dn\u00fd sm\u011br.<\/p>\n<p>Takov\u00e9to vektorov\u00e9 operace se samoz\u0159ejm\u011b daj\u00ed kombinovat, viz obr\u00e1zek 1 vpravo. Ten\u00a0se d\u00e1 interpretovat jako odpov\u011b\u010f na dotaz<em>\u00a0\u201eKing\u201c se m\u00e1 ke \u201ekings\u201c jako se m\u00e1 \u201equeen\u201c k? <\/em>Spr\u00e1vnou odpov\u011bd\u00ed je samoz\u0159ejm\u011b \u201equeens\u201c. Z\u00edsk\u00e1me ji tak, \u017ee zahled\u00e1me\u00a0slovo, jeho\u017e vektorov\u00e1 reprezentace m\u00e1 nejbl\u00ed\u017ee vektoru vec(\u201ekings\u201c) \u2013 vec(\u201eking\u201c) + vec(\u201equeen\u201c). Vhodnou metrikou vzd\u00e1lenosti vektor\u016f je nap\u0159\u00edklad <a title=\"euk\" href=\"http:\/\/en.wikipedia.org\/wiki\/Cosine_similarity\">kosinov\u00e1\u00a0vzd\u00e1lenost<\/a>.\u00a0Pokud m\u00e1me\u00a0dostate\u010dn\u011b velk\u00fd a reprezentativn\u00ed tr\u00e9novac\u00ed korpus, m\u016f\u017eeme se pt\u00e1t i na relativn\u011b slo\u017eit\u00e9 dotazy, jako <em>Kdo je prezidentem Mexika?<\/em> P\u0159\u00edkladem vektorov\u00fdch operac\u00ed, kter\u00e9 by vedly k v\u00fdsledku m\u016f\u017ee b\u00fdt <em>vec<\/em>(<em>\u201e<\/em>Obama\u201c) \u2013 vec(<em>\u201e<\/em>USA\u201c) + vec(<em>\u201e<\/em>Mexico\u201c)<span style=\"line-height: 1.5;\">.<\/span><\/p>\n<p>Zpracov\u00e1n\u00ed velk\u00fdch korpus\u016f pomoc\u00ed p\u016fvodn\u00edho algoritmu v\u0161ak bylo v\u00fdpo\u010detn\u011b p\u0159\u00edli\u0161 n\u00e1ro\u010dn\u00e9. Proto vznikly dva zjednodu\u0161en\u00e9 modely, pomoc\u00ed kter\u00fdch je mo\u017en\u00e9 vygenerovat slovn\u00ed vektory podobn\u00fdch vlastnost\u00ed mnohem efektivn\u011bji \u2013\u00a0<a title=\"C\" href=\"http:\/\/arxiv.org\/pdf\/1301.3781.pdf\">CBOW a Skip-gram<\/a>.\u00a0CBOW model je jednoduch\u00e1 dop\u0159edn\u00e1 neuronov\u00e1 s\u00ed\u0165, kter\u00e1 predikuje slovo v korpusu na z\u00e1klad\u011b sousedn\u00edch slov. Model Skip-gram naopak predikuje okoln\u00ed slova ze slova, kter\u00e9 le\u017e\u00ed mezi nimi. Odkazy na r\u016fzn\u00e9 implementace t\u011bchto dvou model\u016f lze naj\u00edt na domovsk\u00e9 str\u00e1nce projektu word2vec: <a href=\"https:\/\/code.google.com\/p\/word2vec\/\">https:\/\/code.google.com\/p\/word2vec\/<\/a>.<\/p>\n<p>Po velk\u00e9m \u00fasp\u011bchu t\u011bchto model\u016f vznikla \u0159ada dal\u0161\u00edch p\u0159\u00edstup\u016f, jejich\u017e c\u00edlem bylo tak\u00e9\u00a0nalezen\u00ed vektorov\u00e9 reprezentace slov podobn\u00fdch vlastnost\u00ed. Nejv\u00fdrazn\u011bj\u0161\u00edm z nich je z\u0159ejm\u011b stanfordsk\u00fd model <a title=\"GloVe\" href=\"http:\/\/nlp.stanford.edu\/projects\/glove\/\">GloVe<\/a>. \u017d\u00e1dn\u00fd z t\u011bchto model\u016f\u00a0v\u0161ak kvalitativn\u011b v\u00fdrazn\u011b word2vec nep\u0159evy\u0161uje, tak\u017ee nem\u00e1 smysl se jimi v tomto \u010dl\u00e1nku zab\u00fdvat.<\/p>\n<p>Z\u00e1jemci si m\u016f\u017eou vyzkou\u0161et on-line word2vec model napo\u010d\u00edtan\u00fd nad anglickou wikipedi\u00ed nap\u0159\u00edklad na adrese\u00a0<a href=\"http:\/\/deeplearner.fz-qqq.net\">http:\/\/deeplearner.fz-qqq.net<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Kdy\u017e psal Tom\u00e1\u0161 Mikolov svoji diserta\u010dn\u00ed pr\u00e1ci, jist\u011b netu\u0161il, jak\u00fd rozruch jej\u00ed d\u016fsledky v celosv\u011btov\u00e9 NLP komunit\u011b vyvolaj\u00ed. Pr\u00e1ce se zab\u00fdvala jazykov\u00fdmi modely postaven\u00fdmi na neuronov\u00fdch s\u00edt\u00edch. Jazykov\u00e9 modely, kter\u00e9 se b\u011b\u017en\u011b vyu\u017e\u00edvaj\u00ed p\u0159i strojov\u00e9m zpracov\u00e1n\u00ed \u0159e\u010di nebo automatick\u00e9m p\u0159ekladu, jsou matematick\u00e9 modely, umo\u017e\u0148uj\u00edc\u00ed predikovat slova n\u00e1sleduj\u00edc\u00ed po n\u011bjak\u00e9 posloupnosti slov tak, aby vznikl\u00fd text [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[10],"tags":[],"class_list":["post-234","post","type-post","status-publish","format-standard","hentry","category-nlp"],"_links":{"self":[{"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/posts\/234","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/comments?post=234"}],"version-history":[{"count":21,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/posts\/234\/revisions"}],"predecessor-version":[{"id":433,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/posts\/234\/revisions\/433"}],"wp:attachment":[{"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/media?parent=234"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/categories?post=234"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mlguru.com\/cs\/wp-json\/wp\/v2\/tags?post=234"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}