Prior to this commit, we were somewhat naively rendering Hash style data
attributes in our ERB templates. By rendering each hash attribute
separately, we were rendering characters that could break the
javascript (e.g. double hack or backslash `"` or `\`).
By moving to this view_object rendering, we leverage Rails's `to_json`
behavior to ensure properly escaped values. As part of this exercise, I
generalized the method to allow for other places to benefit from this
behavior.
This generalization also helps ensure that we have a more conformant
rendering (e.g. we should always have an :id, :className, and :name
value in our data-info hash).
_Note: I've updated the user's names for Cypress tests as they are more
likely to catch the particular issue than anything else. I assume that
I'm going to break some cypress tests and will need some help fixing
them._
Closes#15916, #14704
Supersedes #15983
How to test locally:
Assuming you have seeded database (e.g. `rails db:seed`), checkout the
"main" branch. Then in `rails console` find a user that's written articles:
```ruby
user = Article.last.user
user.update(name: "\\: #{user.name}")
user.articles.each(&:save)
```
Now, again on the "main" branch, start your application (e.g.,
`bin/startup`).
Then get a logged in and a logged out browser session going. Open your
web inspector and open console. Then go to the local instances homepage
(e.g., http://localhost:3000) and look for JS errors.
On the main branch, you should see an exception around
`JSON.parse(button.data.info)` (assuming that the `user`'s article is
rendered on the homepage).
Then go to the user's page (e.g. https://localhost:3000/:user-slug) and
look for JS parse errors.
On this PR's branch (e.g.,
`jeremyf/take-two-at-resolving-gh-15916`)
you shouldn't see those console errors.
More importantly, the Follow buttons should work.
* Tidying up and documenting Tag model
Prior to this commit we had a custom `where(alias_for: [nil, ""])`
call. That call highlighted that we lacked a term for a Tag that was
not an alias. As part of this commit, I named that a "concrete" tag.
Further, I added scopes to assist in helping "name" those concepts.
This commit also adds a data migration and utilization of
StringAttributeCleaner to hopefully get away from `alias_for == ""`
situations.
As of writing this commit <2022-01-04 Tue 17:22 UTC>, in DEV.to we had 5
tags with `alias_for == ""`:
- actionshackathon21
- regex
- atlashackathon
- hotwire
- foremfest
In https://dev.to/admin/blazer I ran the following:
```sql
SELECT name FROM tags WHERE alias_for = ''
```
* Renaming concrete to direct
* test: whitespace unicode characters cannot be used as titles or tags
* feat: add localized error message
* refactor: use localized error message
* fix: whitespace unicode characters cannot be used as titles or tags
* chore: fix locale after merge
* refactor: fix indentation
* Fix spelling of prohibited in method names
Additionally, rubocop removed a redundant user_id validation since
Article belongs to user.
* Fix spelling
Missed one method call on the same line (fixed the first of two
instances needing changes)
* add failing test cases
The reorganization to remove let was due to a limit on nesting rspec
contexts (it inside context inside describe inside describe when
trying to use different titles in let blocks in contexts)
The first test was clarified (the "U+202D" string that looks like the
code for a unicode point is valid, but the character \u202d is
expected to be invalid
The remaining two tests are based on the feedback I'd given in the PR,
failing because we don't assert title is present after removing
unicode whitespace, and we don't actually set the title (gsub is
non-destructive, returning a value).
* Set title to sanitized title
contains_prohibitied_unicode_characters? was using == (which is not a
good match for strings to regexes), use =~ instead
Change invalid example characters in titles from \u202d (bidi override) to
\u200a (hair space)
Assert that the empty title is blank and can't be blank (even though
it contains no prohibitd characters after replacement)
* change the definition of prohibited characters
From the description, it looks like "unicode space characters" was the
desired rejection set. It was unclear what the existing regex was
matching on (I couldn't get the expected examples to match correctly).
Given the intent, I select "unicode space property, except the ascii
space character", which may _also_ be incorrect but passed the tests.
* Remove now-invalid expectation for presence of user id
Since this was redundant (belongs_to user), we no longer will validate
presence of user id, and we should not test for it.
* Only reject bidirectional text controls
The original issue pointed to the BIDI controls as problematic.
While they called out other "whitespace" characters, as we accept a
wider range of input languages, the need to handle non-printing
punctuation (for example a group separating space for some asian
languages).
It's possible a wider list of characters should be added - if that's
the case I suggest this regex and the sanitization be moved to a
standalone class, and each of the recommendations in
https://www.w3.org/TR/unicode-xml/#Charlist be checked for
applicability.
* Check for blank title after sanitizing disallowed characters
Since validate_title modifies the title, and doesn't set error
messages on the model itself, check for non-empty title _after_
potentially removing any disallowed characters.
* Remove invalid characters before validation
Remove the validate_title and contains_prohibited_characters methods,
and center all logic on the remove prohibitied unicode characters
method.
Remove unneeded and unused arguments (input string is always title,
replacement is always removal/empty string).
* handle case when validating null title
If the title's nil, we don't want to call match? since it will fail.
Title will sometimes be nil when validating.
Co-authored-by: Ben Halpern <bendhalpern@gmail.com>
Co-authored-by: Dan Uber <dan@forem.com>
* Favoring single SQL and computation over enum
Prior to this commit, we ran a SQL statement to generate a list of
articles then applied some minor randomization.
With this commit, I'm removing the randomization in favor of expected
values, and collapsing the result set into a single inline SQL and some
post query simple arithmatic.
Let's check my math. For each article we sum:
* `article.score`
* `article.comments_count` * 14
* a random number between 0 and 5 (`rand(6)`) with expected value of 2.5
* the tag's (taggings_count + 1) / 2
The new SQL query sums the `article.score` and the
`article.comments_count` * 14. I then reduce the remainder of the
equation. From "for each article sum `(rand(5) + (taggings_count + 1) /
2)`" we have the following:
`article_count * (2.5 + (taggings_count + 1) / 2)`
Which is equivalent to:
`article_count * ((taggings_count + 1 + (2.5 * 2)) / 2)`
Which is equivalent to:
`article_count * ((taggings_count + 6) / 2)`
This change reduces computation time by favoring expected values.
* Fixing broken behavior and adding comments
* Remove Connect
* Remove more Connect specs
* Remove a lot more Connect code
* 🚮
* It all has to go
* Explicitly add httpclient
* Update application layout
* Remove messages association from User
* Start fixing specs
* reintroduce util function and refactor references
* Remove Connect Cypress test
* Fix more specs
* Remove Connect from listings
* Ignore contact_via_connect column on listings
* Remove contact_via_connect usages
* Ignore mod_chat_channel_id on tags
* Drop Connect tables
* Remove email_connect_messages from user notification settings
* Re-add httpclient 2.8.3
This was mistakenly removed as a merge conflict
* Don't need to exclude removed chat channel file
* Remove unneeded style for chat channels
* Remove unneeded channel list prop type
* Remove chat channels index/connect-link from getPageEntries
* Re-add comment from httpclient in Gemfile
* Remove connect references from mailers
Tag Moderators no longer have a chat channel
No longer will users be notified about new messages (there won't be
any)
No longer will users be notified about channel invites (you can't
invite anyone anymore)
* Don't configure Pusher and remove PUSHER_* from .env_sample
since it's removed from gemfile, the Pusher constant will not resolve, if this is
configured in the environment variables we'll fail to boot.
Co-authored-by: Suzanne Aitchison <suzanne@forem.com>
Co-authored-by: Dan Uber <dan@forem.com>
* show top tags when search is empty
* add cypress tests
* nudge travis
* tweak to e2e test
* show a 'top tags' heading
* only suggest tags that are supported
* Add supported scope to Tag
Co-authored-by: Michael Kohl <citizen428@forem.com>
* Relax tag validation restriction to permit alphanumeric strings
Non-word symbols like musical notes are still blocked, but most
languages should work.
* Handle redirect after tag edit
The URI.parse(tag.name) was not updated in #13720
Move reused logic into tag_path method in URL module.
* FactoryBot generated tags are non-ascii
This should help expose any cases where we're mistakenly counting on
ascii-only content in tags by causing all generated tags without a
specified name to be of the form `tag你好#{n}` instead of `tag#{n}`.
* Revert "FactoryBot generated tags are non-ascii"
This reverts commit ed1035cd1e91619b2c1599e521b6ddf962c97f80.
It was useful to see how many places the path knowledge is in the
tests (a lot of requests have GET /t/tag.name and need to have GET
URL.tag_path instead).
https://app.travis-ci.com/github/forem/forem/builds/237542600 has the
results.
Reverting because those test-side changes seem like a secondary
concern to the change in this PR (I'll file a follow on issue)
* add: article cache on update tag
* minor styling fix
* add: published scope to fetch articles
* update: tag_spec:125 to not fetch articles
* modify: tag.rb:bust_cache to delete server cache
* add: tests to verify cache delete
* Step one in populating the reading list with PG
This first attempt tries to recycle the `Search::ArticleSerializer` which is only
used in input in ES, but we're using it in output in PG.
For this reason it's currently 15.55x times slower
* Serialize only what is requested by the frontend
`Search::ArticleSerializer` which is only used in ES in the indexing step aims
to add as much info as possible for broader purposes, in this case
(with serialization in output) we should aim to save only what's requested from
the frontend.
* Optimize selection of articles columns
* Select only needed columns for users
* Compute total of reading list items
* Attach the basic filtering based on PG on the search controller
* Restructure in methods
* Add tags support
* Use LIKE on articles.cached_tag_list
* Fix tags as nil
* Fix default pagination
* Add optional FTS for reading list
* Reworded the tags comment explaining why
* Add index to reactions.status
* Fix total counter in Preact readingList component
* Fix total count in reading list backend search
* Add GIN index to articles.cached_tag_list
* Add service tests
* Add search request specs
* Added missing early return
* Update spec/requests/search_spec.rb
Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
* Extract MAX_PER_PAGE constant and add comments
Co-authored-by: Julianna Tetreault <32834804+juliannatetreault@users.noreply.github.com>
* Add Search::Postgres::Tag to retrieve tags for autocompletion
* Add Search::Postgres::Tag in SearchController behind a feature flag
* Update spec/services/search/postgres/tag_spec.rb
Co-authored-by: Michael Kohl <citizen428@dev.to>
* Add index to tags.supported
* Fix hotness_score ordering
Co-authored-by: Michael Kohl <citizen428@dev.to>
* Don't add to mailchimp for N/A circumstance
* Use class << self over self. methods
* Refactor and make add/remove tag mod idempotent
* Add missing param wiki_body_markdown
* Add errors_as_sentence b/c Tag not inheriting from AppRecord
* Refactor and Crayonsify tag page
* Use proper for attr to make label checkable
* Crayonsify moderator section
* Make placeholder white
* Remove remote: false for sync form submit
* Fix more merge conflicts
* Minor styling adjustments
* Fix color field value
* Don't add to Mailchimp community mod list if not enabled
* Add tests for new tag mod routes
* Add missing line oops
* Use self.method over class << self for new Rubyists
* Use more efficient query for getting tag mods
* Use parentheses to follow convention
* Use cleaner way of routing and controller
* Update tests to match new configuration
* Add missing FKs to BackupData BadgeAchievements BanishedUsers BufferUpdates
* Add relations and specs
* Fix specs
* Trigger Travis CI
* Remove traces of backup_data has it has been removed on master
* Fix trailing whitespace
* Enable new Rails/* cops and use autocorrect on them
* Fixed Rails/PluckInWhere leftovers
* Fix Rails/DefaultScope
* Enable and fix Rails/PluckId
* Fix manual mistake with forcing autocorrection on Rails/PluckId
* Apply PR feedback to remove Rails/PluckId inline disables
* Apply PR feedback to get rid of Rails/PluckInWhere inline
* Stop allowing accented characters in tags
This commit will add strict frontend and backend validation for tags
disallowing accented characters.
* Update tags validation error message
* Add more character sets to tag validation spec
* Add comment referencing Fastly gem
* Create purgable concern and include it
* Typo: deprecate --> deprecated
* Include Purgable on the Tag model
* Rename Purgable to Purgeable
* Reorganize fastly and service methods in Purgeable
* Add and fix specs
* Add development env guards
* Custom social preview template for tags
* Added index on tags social_preview_template + small refactoring
* Add checks if index exists to tags migration
* Index tags to Elasticsearch after they are created
* add rules_html to Tag index since we show it in search drop down
* change get method to find_document to make it more explicit
