Getting started with spaCy


  • Doc: Sequence of Token
  • Token: An individual token — i.e. a word, punctuation symbol, whitespace, etc.
  • Span: A slice from a Doc object.
  • Lexeme: A Lexeme has no string context – it’s a word type, as opposed to a word token. It therefore has no part-of-speech tag, dependency parse, or lemma


The tokenizer processes the text from left to right. On each substring, it performs two checks:

  • Does the substring match a tokenizer exception rule?
  • Can a prefix, suffix or infix be split off?

If there’s a match, the rule is applied and the tokenizer continues its loop.

Named Entity Recognition:

  • Named entities are available as the ents property of a Doc
  • The standard way to access entity annotations is doc.ents , which produces a sequence of span.
  • We can also access token entity annotations using the token.ent_iob and token.ent_type attributes. token.ent_iob indicates whether an entity starts, continues or ends on the tag.


  • I – Token is inside an entity.
  • O – Token is outside an entity.
  • B – Token is the beginning of an entity