建立倒排索引
有两个doc,
doc1:I really liked my small dogs, and I think my mom also liked them.
doc2:He never liked any dogs, so I hope that my mom will not expect me to liked him.
分词,初步的倒排索引的建立
worddoc1doc2I**really*liked**my**small*dogs*and*think*mom**also*them*He*never*any*so*hope*that*will*not*expect*me*to*him*演示了一下倒排索引最简单的建立的一个过程
进行搜索mother like little dog
分词
mother
like
little
dog
分词后去倒排索引去匹配,没有任何结果
这个不是我们想要的搜索结果,因为在我们看来,mother和mom是同义词,都是妈妈的意思。like和liked没有区别吗,都是喜欢的意思,只不过一个是现在时,一个是过去时。little和small没有区别,是同义词,都是小的意思。dog和dogs也没有区别,都是狗,只不过一个是单数,一个是复数。
建立索引的时候,除了创建倒排索引,还会执行normalization操作normalization:就是说对拆分出的各个单词进行相应的处理,以提升后面搜索的时候能够搜索到相关联的文档的概率
比如时态的转换,单复数的转换,同义词的转换,大小写的转换
mom —> mother
liked —> like
small —> little
dogs —> dog
重新建立倒排索引,加入normalization,再次用mother like little dog搜索,就可以搜索到了
worddoc1doc2normalizationI**really*like**liked --> likemy**little*small --> littledog*dogs --> dogand*think*mom**mother ->momalso*them*He*never*any*so*hope*that*will*not*expect*me*to*him*mother like little dog,分词,经过normalization操作
统一变为
mother --> mom
liked --> like
small --> little
dogs --> dog
所以doc1和doc2都能搜索出来
doc1:I really liked my small dogs, and I think my mom also liked them.
doc2:He never liked any dogs, so I hope that my mom will not expect me to liked him.