docbrown/app/services/languages/detection.rb
Ben Halpern 8335a37315
Language detection with CLD3 (#19756)
* Language detection with CLD3 POC

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update app/models/article.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Clean up tests

* Clean up tests

* Move language detection to service

* rubocop

* Update app/services/languages/detection.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* More linting cleanup

* Remove allow_any_instance_of

* Update tests to better exercise different paths

* Fix flaky spec

* Update spec/services/languages/detection_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/models/article_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/services/languages/detection_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/services/languages/detection_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/services/languages/detection_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update spec/services/languages/detection_spec.rb

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-07-17 08:57:20 -04:00

22 lines
474 B
Ruby

module Languages
class Detection
attr_reader :text
PROBABILITY_THRESHOLD = 0.5
def self.call(...)
new(...).call
end
def initialize(text)
@text = text
end
def call(identifier: CLD3::NNetLanguageIdentifier.new(0, 1000))
language_outcome = identifier.find_language(text)
return unless language_outcome.probability > PROBABILITY_THRESHOLD && language_outcome.reliable?
language_outcome.language
end
end
end