Browse Source

Revamp post filtering system (#18058)

* Add model for custom filter keywords

* Use CustomFilterKeyword internally

Does not change the API

* Fix /filters/edit and /filters/new

* Add migration tests

* Remove whole_word column from custom_filters (covered by custom_filter_keywords)

* Redesign /filters

Instead of a list, present a card that displays more information and handles
multiple keywords per filter.

* Redesign /filters/new and /filters/edit to add and remove keywords

This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.

* Add /api/v2/filters to edit filter with multiple keywords

Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
  `keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`

API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
  `keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
  `keywords_attributes` can also be passed to edit, delete or add keywords in
   one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
   filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword

* Change from `irreversible` boolean to `action` enum

* Remove irrelevent `irreversible_must_be_within_context` check

* Fix /filters/new and /filters/edit with update for filter_action

* Fix Rubocop/Codeclimate complaining about task names

* Refactor FeedManager#phrase_filtered?

This moves regexp building and filter caching to the `CustomFilter` class.

This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.

* Perform server-side filtering and output result in REST API

* Fix numerous filters_changed events being sent when editing multiple keywords at once

* Add some tests

* Use the new API in the WebUI

- use client-side logic for filters we have fetched rules for.
  This is so that filter changes can be retroactively applied without
  reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
  (e.g. network error, or initial timeline loading)

* Minor optimizations and refactoring

* Perform server-side filtering on the streaming server

* Change the wording of filter action labels

* Fix issues pointed out by linter

* Change design of “Show anyway” link in accordence to review comments

* Drop “irreversible” filtering behavior

* Move /api/v2/filter_keywords to /api/v1/filters/keywords

* Rename `filter_results` attribute to `filtered`

* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer

* Fix systemChannelId value in streaming server

* Simplify code by removing client-side filtering code

The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
main
Claire 1 month ago committed by GitHub
parent
commit
02851848e9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
  1. 18
      .circleci/config.yml
  2. 2
      Gemfile
  3. 2
      Gemfile.lock
  4. 50
      app/controllers/api/v1/filters/keywords_controller.rb
  5. 35
      app/controllers/api/v1/filters_controller.rb
  6. 48
      app/controllers/api/v2/filters_controller.rb
  7. 12
      app/controllers/filters_controller.rb
  8. 26
      app/javascript/mastodon/actions/filters.js
  9. 11
      app/javascript/mastodon/actions/importer/index.js
  10. 12
      app/javascript/mastodon/actions/importer/normalizer.js
  11. 13
      app/javascript/mastodon/actions/notifications.js
  12. 4
      app/javascript/mastodon/actions/streaming.js
  13. 21
      app/javascript/mastodon/components/status.js
  14. 17
      app/javascript/mastodon/components/status_action_bar.js
  15. 3
      app/javascript/mastodon/features/ui/index.js
  16. 34
      app/javascript/mastodon/reducers/filters.js
  17. 51
      app/javascript/mastodon/selectors/index.js
  18. 1
      app/javascript/packs/public.js
  19. 33
      app/javascript/styles/mastodon/admin.scss
  20. 15
      app/javascript/styles/mastodon/components.scss
  21. 31
      app/javascript/styles/mastodon/forms.scss
  22. 30
      app/lib/feed_manager.rb
  23. 13
      app/models/concerns/account_interactions.rb
  24. 87
      app/models/custom_filter.rb
  25. 34
      app/models/custom_filter_keyword.rb
  26. 5
      app/presenters/filter_result_presenter.rb
  27. 24
      app/presenters/status_relationships_presenter.rb
  28. 9
      app/serializers/rest/filter_keyword_serializer.rb
  29. 6
      app/serializers/rest/filter_result_serializer.rb
  30. 8
      app/serializers/rest/filter_serializer.rb
  31. 9
      app/serializers/rest/status_serializer.rb
  32. 26
      app/serializers/rest/v1/filter_serializer.rb
  33. 16
      app/views/filters/_fields.html.haml
  34. 32
      app/views/filters/_filter.html.haml
  35. 33
      app/views/filters/_filter_fields.html.haml
  36. 8
      app/views/filters/_keyword_fields.html.haml
  37. 2
      app/views/filters/edit.html.haml
  38. 17
      app/views/filters/index.html.haml
  39. 4
      app/views/filters/new.html.haml
  40. 11
      config/locales/en.yml
  41. 10
      config/locales/simple_form.en.yml
  42. 9
      config/routes.rb
  43. 13
      db/migrate/20220613110628_create_custom_filter_keywords.rb
  44. 34
      db/migrate/20220613110711_migrate_custom_filters.rb
  45. 20
      db/migrate/20220613110834_add_action_to_custom_filters.rb
  46. 20
      db/post_migrate/20220613110802_remove_whole_word_from_custom_filters.rb
  47. 20
      db/post_migrate/20220613110903_remove_irreversible_from_custom_filters.rb
  48. 15
      db/schema.rb
  49. 18
      lib/tasks/tests.rake
  50. 2
      package.json
  51. 142
      spec/controllers/api/v1/filters/keywords_controller_spec.rb
  52. 27
      spec/controllers/api/v1/filters_controller_spec.rb
  53. 52
      spec/controllers/api/v1/statuses_controller_spec.rb
  54. 121
      spec/controllers/api/v2/filters_controller_spec.rb
  55. 4
      spec/fabricators/custom_filter_keyword_fabricator.rb
  56. 32
      spec/lib/feed_manager_spec.rb
  57. 4
      spec/models/custom_filter_keyword_spec.rb
  58. 29
      spec/presenters/status_relationships_presenter_spec.rb
  59. 90
      streaming/index.js
  60. 67
      yarn.lock

18
.circleci/config.yml

@ -133,6 +133,12 @@ jobs:
- run:
command: ./bin/rails tests:migrations:populate_v2_4
name: Populate database with test data
- run:
command: ./bin/rails db:migrate VERSION=20180707154237
name: Run migrations up to v2.4.3
- run:
command: ./bin/rails tests:migrations:populate_v2_4_3
name: Populate database with test data
- run:
command: ./bin/rails db:migrate
name: Run all remaining migrations
@ -167,14 +173,22 @@ jobs:
- run:
command: ./bin/rails tests:migrations:populate_v2_4
name: Populate database with test data
- run:
command: ./bin/rails db:migrate VERSION=20180707154237
name: Run migrations up to v2.4.3
environment:
SKIP_POST_DEPLOYMENT_MIGRATIONS: true
- run:
command: ./bin/rails tests:migrations:populate_v2_4_3
name: Populate database with test data
- run:
command: ./bin/rails db:migrate
name: Run all pre-deployment migrations
name: Run all remaining pre-deployment migrations
environment:
SKIP_POST_DEPLOYMENT_MIGRATIONS: true
- run:
command: ./bin/rails db:migrate
name: Run all post-deployment remaining migrations
name: Run all post-deployment migrations
- run:
command: ./bin/rails tests:migrations:check_database
name: Check migration result

2
Gemfile

@ -153,3 +153,5 @@ gem 'concurrent-ruby', require: false
gem 'connection_pool', require: false
gem 'xorcist', '~> 1.1'
gem 'cocoon', '~> 1.2'

2
Gemfile.lock

@ -163,6 +163,7 @@ GEM
elasticsearch-dsl
chunky_png (1.4.0)
climate_control (0.2.0)
cocoon (1.2.15)
coderay (1.1.3)
color_diff (0.1)
concurrent-ruby (1.1.10)
@ -746,6 +747,7 @@ DEPENDENCIES
charlock_holmes (~> 0.7.7)
chewy (~> 7.2)
climate_control (~> 0.2)
cocoon (~> 1.2)
color_diff (~> 0.1)
concurrent-ruby
connection_pool

50
app/controllers/api/v1/filters/keywords_controller.rb

@ -0,0 +1,50 @@
# frozen_string_literal: true
class Api::V1::Filters::KeywordsController < Api::BaseController
before_action -> { doorkeeper_authorize! :read, :'read:filters' }, only: [:index, :show]
before_action -> { doorkeeper_authorize! :write, :'write:filters' }, except: [:index, :show]
before_action :require_user!
before_action :set_keywords, only: :index
before_action :set_keyword, only: [:show, :update, :destroy]
def index
render json: @keywords, each_serializer: REST::FilterKeywordSerializer
end
def create
@keyword = current_account.custom_filters.find(params[:filter_id]).keywords.create!(resource_params)
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def show
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def update
@keyword.update!(resource_params)
render json: @keyword, serializer: REST::FilterKeywordSerializer
end
def destroy
@keyword.destroy!
render_empty
end
private
def set_keywords
filter = current_account.custom_filters.includes(:keywords).find(params[:filter_id])
@keywords = filter.keywords
end
def set_keyword
@keyword = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account }).find(params[:id])
end
def resource_params
params.permit(:keyword, :whole_word)
end
end

35
app/controllers/api/v1/filters_controller.rb

@ -8,21 +8,32 @@ class Api::V1::FiltersController < Api::BaseController
before_action :set_filter, only: [:show, :update, :destroy]
def index
render json: @filters, each_serializer: REST::FilterSerializer
render json: @filters, each_serializer: REST::V1::FilterSerializer
end
def create
@filter = current_account.custom_filters.create!(resource_params)
render json: @filter, serializer: REST::FilterSerializer
ApplicationRecord.transaction do
filter_category = current_account.custom_filters.create!(resource_params)
@filter = filter_category.keywords.create!(keyword_params)
end
render json: @filter, serializer: REST::V1::FilterSerializer
end
def show
render json: @filter, serializer: REST::FilterSerializer
render json: @filter, serializer: REST::V1::FilterSerializer
end
def update
@filter.update!(resource_params)
render json: @filter, serializer: REST::FilterSerializer
ApplicationRecord.transaction do
@filter.update!(keyword_params)
@filter.custom_filter.assign_attributes(filter_params)
raise Mastodon::ValidationError, I18n.t('filters.errors.deprecated_api_multiple_keywords') if @filter.custom_filter.changed? && @filter.custom_filter.keywords.count > 1
@filter.custom_filter.save!
end
render json: @filter, serializer: REST::V1::FilterSerializer
end
def destroy
@ -33,14 +44,22 @@ class Api::V1::FiltersController < Api::BaseController
private
def set_filters
@filters = current_account.custom_filters
@filters = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account })
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
@filter = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account: current_account }).find(params[:id])
end
def resource_params
params.permit(:phrase, :expires_in, :irreversible, :whole_word, context: [])
end
def filter_params
resource_params.slice(:expires_in, :irreversible, :context)
end
def keyword_params
resource_params.slice(:phrase, :whole_word)
end
end

48
app/controllers/api/v2/filters_controller.rb

@ -0,0 +1,48 @@
# frozen_string_literal: true
class Api::V2::FiltersController < Api::BaseController
before_action -> { doorkeeper_authorize! :read, :'read:filters' }, only: [:index, :show]
before_action -> { doorkeeper_authorize! :write, :'write:filters' }, except: [:index, :show]
before_action :require_user!
before_action :set_filters, only: :index
before_action :set_filter, only: [:show, :update, :destroy]
def index
render json: @filters, each_serializer: REST::FilterSerializer, rules_requested: true
end
def create
@filter = current_account.custom_filters.create!(resource_params)
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def show
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def update
@filter.update!(resource_params)
render json: @filter, serializer: REST::FilterSerializer, rules_requested: true
end
def destroy
@filter.destroy!
render_empty
end
private
def set_filters
@filters = current_account.custom_filters.includes(:keywords)
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
end
def resource_params
params.permit(:title, :expires_in, :filter_action, context: [], keywords_attributes: [:id, :keyword, :whole_word, :_destroy])
end
end

12
app/controllers/filters_controller.rb

@ -4,16 +4,16 @@ class FiltersController < ApplicationController
layout 'admin'
before_action :authenticate_user!
before_action :set_filters, only: :index
before_action :set_filter, only: [:edit, :update, :destroy]
before_action :set_body_classes
def index
@filters = current_account.custom_filters.order(:phrase)
@filters = current_account.custom_filters.includes(:keywords).order(:phrase)
end
def new
@filter = current_account.custom_filters.build
@filter = current_account.custom_filters.build(action: :warn)
@filter.keywords.build
end
def create
@ -43,16 +43,12 @@ class FiltersController < ApplicationController
private
def set_filters
@filters = current_account.custom_filters
end
def set_filter
@filter = current_account.custom_filters.find(params[:id])
end
def resource_params
params.require(:custom_filter).permit(:phrase, :expires_in, :irreversible, :whole_word, context: [])
params.require(:custom_filter).permit(:title, :expires_in, :filter_action, context: [], keywords_attributes: [:id, :keyword, :whole_word, :_destroy])
end
def set_body_classes

26
app/javascript/mastodon/actions/filters.js

@ -1,26 +0,0 @@
import api from '../api';
export const FILTERS_FETCH_REQUEST = 'FILTERS_FETCH_REQUEST';
export const FILTERS_FETCH_SUCCESS = 'FILTERS_FETCH_SUCCESS';
export const FILTERS_FETCH_FAIL = 'FILTERS_FETCH_FAIL';
export const fetchFilters = () => (dispatch, getState) => {
dispatch({
type: FILTERS_FETCH_REQUEST,
skipLoading: true,
});
api(getState)
.get('/api/v1/filters')
.then(({ data }) => dispatch({
type: FILTERS_FETCH_SUCCESS,
filters: data,
skipLoading: true,
}))
.catch(err => dispatch({
type: FILTERS_FETCH_FAIL,
err,
skipLoading: true,
skipAlert: true,
}));
};

11
app/javascript/mastodon/actions/importer/index.js

@ -5,6 +5,7 @@ export const ACCOUNTS_IMPORT = 'ACCOUNTS_IMPORT';
export const STATUS_IMPORT = 'STATUS_IMPORT';
export const STATUSES_IMPORT = 'STATUSES_IMPORT';
export const POLLS_IMPORT = 'POLLS_IMPORT';
export const FILTERS_IMPORT = 'FILTERS_IMPORT';
function pushUnique(array, object) {
if (array.every(element => element.id !== object.id)) {
@ -28,6 +29,10 @@ export function importStatuses(statuses) {
return { type: STATUSES_IMPORT, statuses };
}
export function importFilters(filters) {
return { type: FILTERS_IMPORT, filters };
}
export function importPolls(polls) {
return { type: POLLS_IMPORT, polls };
}
@ -61,11 +66,16 @@ export function importFetchedStatuses(statuses) {
const accounts = [];
const normalStatuses = [];
const polls = [];
const filters = [];
function processStatus(status) {
pushUnique(normalStatuses, normalizeStatus(status, getState().getIn(['statuses', status.id])));
pushUnique(accounts, status.account);
if (status.filtered) {
status.filtered.forEach(result => pushUnique(filters, result.filter));
}
if (status.reblog && status.reblog.id) {
processStatus(status.reblog);
}
@ -80,6 +90,7 @@ export function importFetchedStatuses(statuses) {
dispatch(importPolls(polls));
dispatch(importFetchedAccounts(accounts));
dispatch(importStatuses(normalStatuses));
dispatch(importFilters(filters));
};
}

12
app/javascript/mastodon/actions/importer/normalizer.js

@ -42,6 +42,14 @@ export function normalizeAccount(account) {
return account;
}
export function normalizeFilterResult(result) {
const normalResult = { ...result };
normalResult.filter = normalResult.filter.id;
return normalResult;
}
export function normalizeStatus(status, normalOldStatus) {
const normalStatus = { ...status };
normalStatus.account = status.account.id;
@ -54,6 +62,10 @@ export function normalizeStatus(status, normalOldStatus) {
normalStatus.poll = status.poll.id;
}
if (status.filtered) {
normalStatus.filtered = status.filtered.map(normalizeFilterResult);
}
// Only calculate these values when status first encountered and
// when the underlying values change. Otherwise keep the ones
// already in the reducer

13
app/javascript/mastodon/actions/notifications.js

@ -12,10 +12,8 @@ import { saveSettings } from './settings';
import { defineMessages } from 'react-intl';
import { List as ImmutableList } from 'immutable';
import { unescapeHTML } from '../utils/html';
import { getFiltersRegex } from '../selectors';
import { usePendingItems as preferPendingItems } from 'mastodon/initial_state';
import compareId from 'mastodon/compare_id';
import { searchTextFromRawStatus } from 'mastodon/actions/importer/normalizer';
import { requestNotificationPermission } from '../utils/notifications';
export const NOTIFICATIONS_UPDATE = 'NOTIFICATIONS_UPDATE';
@ -62,20 +60,17 @@ export function updateNotifications(notification, intlMessages, intlLocale) {
const showInColumn = activeFilter === 'all' ? getState().getIn(['settings', 'notifications', 'shows', notification.type], true) : activeFilter === notification.type;
const showAlert = getState().getIn(['settings', 'notifications', 'alerts', notification.type], true);
const playSound = getState().getIn(['settings', 'notifications', 'sounds', notification.type], true);
const filters = getFiltersRegex(getState(), { contextType: 'notifications' });
let filtered = false;
if (['mention', 'status'].includes(notification.type)) {
const dropRegex = filters[0];
const regex = filters[1];
const searchIndex = searchTextFromRawStatus(notification.status);
if (['mention', 'status'].includes(notification.type) && notification.status.filtered) {
const filters = notification.status.filtered.filter(result => result.filter.context.includes('notifications'));
if (dropRegex && dropRegex.test(searchIndex)) {
if (filters.some(result => result.filter.filter_action === 'hide')) {
return;
}
filtered = regex && regex.test(searchIndex);
filtered = filters.length > 0;
}
if (['follow_request'].includes(notification.type)) {

4
app/javascript/mastodon/actions/streaming.js

@ -21,7 +21,6 @@ import {
updateReaction as updateAnnouncementsReaction,
deleteAnnouncement,
} from './announcements';
import { fetchFilters } from './filters';
import { getLocale } from '../locales';
const { messages } = getLocale();
@ -97,9 +96,6 @@ export const connectTimelineStream = (timelineId, channelName, params = {}, opti
case 'conversation':
dispatch(updateConversations(JSON.parse(data.payload)));
break;
case 'filters_changed':
dispatch(fetchFilters());
break;
case 'announcement':
dispatch(updateAnnouncements(JSON.parse(data.payload)));
break;

21
app/javascript/mastodon/components/status.js

@ -116,6 +116,7 @@ class Status extends ImmutablePureComponent {
state = {
showMedia: defaultMediaVisibility(this.props.status),
statusId: undefined,
forceFilter: undefined,
};
static getDerivedStateFromProps(nextProps, prevState) {
@ -277,6 +278,15 @@ class Status extends ImmutablePureComponent {
this.handleToggleMediaVisibility();
}
handleUnfilterClick = e => {
this.setState({ forceFilter: false });
e.preventDefault();
}
handleFilterClick = () => {
this.setState({ forceFilter: true });
}
_properStatus () {
const { status } = this.props;
@ -328,7 +338,8 @@ class Status extends ImmutablePureComponent {
);
}
if (status.get('filtered') || status.getIn(['reblog', 'filtered'])) {
const matchedFilters = status.get('filtered') || status.getIn(['reblog', 'filtered']);
if (this.state.forceFilter === undefined ? matchedFilters : this.state.forceFilter) {
const minHandlers = this.props.muted ? {} : {
moveUp: this.handleHotkeyMoveUp,
moveDown: this.handleHotkeyMoveDown,
@ -337,7 +348,11 @@ class Status extends ImmutablePureComponent {
return (
<HotKeys handlers={minHandlers}>
<div className='status__wrapper status__wrapper--filtered focusable' tabIndex='0' ref={this.handleRef}>
<FormattedMessage id='status.filtered' defaultMessage='Filtered' />
<FormattedMessage id='status.filtered' defaultMessage='Filtered' />: {matchedFilters.join(', ')}.
{' '}
<button className='status__wrapper--filtered__button' onClick={this.handleUnfilterClick}>
<FormattedMessage id='status.show_filter_reason' defaultMessage='Show anyway' />
</button>
</div>
</HotKeys>
);
@ -496,7 +511,7 @@ class Status extends ImmutablePureComponent {
{media}
<StatusActionBar scrollKey={scrollKey} status={status} account={account} {...other} />
<StatusActionBar scrollKey={scrollKey} status={status} account={account} onFilter={matchedFilters && this.handleFilterClick} {...other} />
</div>
</div>
</HotKeys>

17
app/javascript/mastodon/components/status_action_bar.js

@ -38,6 +38,7 @@ const messages = defineMessages({
admin_account: { id: 'status.admin_account', defaultMessage: 'Open moderation interface for @{name}' },
admin_status: { id: 'status.admin_status', defaultMessage: 'Open this status in the moderation interface' },
copy: { id: 'status.copy', defaultMessage: 'Copy link to status' },
hide: { id: 'status.hide', defaultMessage: 'Hide toot' },
blockDomain: { id: 'account.block_domain', defaultMessage: 'Block domain {domain}' },
unblockDomain: { id: 'account.unblock_domain', defaultMessage: 'Unblock domain {domain}' },
unmute: { id: 'account.unmute', defaultMessage: 'Unmute @{name}' },
@ -76,6 +77,7 @@ class StatusActionBar extends ImmutablePureComponent {
onMuteConversation: PropTypes.func,
onPin: PropTypes.func,
onBookmark: PropTypes.func,
onFilter: PropTypes.func,
withDismiss: PropTypes.bool,
withCounters: PropTypes.bool,
scrollKey: PropTypes.string,
@ -207,6 +209,10 @@ class StatusActionBar extends ImmutablePureComponent {
this.props.onMuteConversation(this.props.status);
}
handleFilter = () => {
this.props.onFilter();
}
handleCopy = () => {
const url = this.props.status.get('url');
const textarea = document.createElement('textarea');
@ -226,6 +232,11 @@ class StatusActionBar extends ImmutablePureComponent {
}
}
handleFilterClick = () => {
this.props.onFilter();
}
render () {
const { status, relationship, intl, withDismiss, withCounters, scrollKey } = this.props;
@ -329,6 +340,10 @@ class StatusActionBar extends ImmutablePureComponent {
<IconButton className='status__action-bar-button' title={intl.formatMessage(messages.share)} icon='share-alt' onClick={this.handleShareClick} />
);
const filterButton = this.props.onFilter && (
<IconButton className='status__action-bar-button' title={intl.formatMessage(messages.hide)} icon='eye' onClick={this.handleFilterClick} />
);
return (
<div className='status__action-bar'>
<IconButton className='status__action-bar-button' title={replyTitle} icon={status.get('in_reply_to_account_id') === status.getIn(['account', 'id']) ? 'reply' : replyIcon} onClick={this.handleReplyClick} counter={status.get('replies_count')} obfuscateCount />
@ -337,6 +352,8 @@ class StatusActionBar extends ImmutablePureComponent {
{shareButton}
{filterButton}
<div className='status__action-bar-dropdown'>
<DropdownMenuContainer
scrollKey={scrollKey}

3
app/javascript/mastodon/features/ui/index.js

@ -13,7 +13,6 @@ import { debounce } from 'lodash';
import { uploadCompose, resetCompose, changeComposeSpoilerness } from '../../actions/compose';
import { expandHomeTimeline } from '../../actions/timelines';
import { expandNotifications } from '../../actions/notifications';
import { fetchFilters } from '../../actions/filters';
import { fetchRules } from '../../actions/rules';
import { clearHeight } from '../../actions/height_cache';
import { focusApp, unfocusApp, changeLayout } from 'mastodon/actions/app';
@ -368,7 +367,7 @@ class UI extends React.PureComponent {
this.props.dispatch(fetchMarkers());
this.props.dispatch(expandHomeTimeline());
this.props.dispatch(expandNotifications());
setTimeout(() => this.props.dispatch(fetchFilters()), 500);
setTimeout(() => this.props.dispatch(fetchRules()), 3000);
this.hotkeys.__mousetrap__.stopCallback = (e, element) => {

34
app/javascript/mastodon/reducers/filters.js

@ -1,10 +1,34 @@
import { FILTERS_FETCH_SUCCESS } from '../actions/filters';
import { List as ImmutableList, fromJS } from 'immutable';
import { FILTERS_IMPORT } from '../actions/importer';
import { Map as ImmutableMap, is, fromJS } from 'immutable';
export default function filters(state = ImmutableList(), action) {
const normalizeFilter = (state, filter) => {
const normalizedFilter = fromJS({
id: filter.id,
title: filter.title,
context: filter.context,
filter_action: filter.filter_action,
expires_at: filter.expires_at ? Date.parse(filter.expires_at) : null,
});
if (is(state.get(filter.id), normalizedFilter)) {
return state;
} else {
return state.set(filter.id, normalizedFilter);
}
};
const normalizeFilters = (state, filters) => {
filters.forEach(filter => {
state = normalizeFilter(state, filter);
});
return state;
};
export default function filters(state = ImmutableMap(), action) {
switch(action.type) {
case FILTERS_FETCH_SUCCESS:
return fromJS(action.filters);
case FILTERS_IMPORT:
return normalizeFilters(state, action.filters);
default:
return state;
}

51
app/javascript/mastodon/selectors/index.js

@ -40,15 +40,15 @@ const toServerSideType = columnType => {
const escapeRegExp = string =>
string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string
const regexFromFilters = filters => {
if (filters.size === 0) {
const regexFromKeywords = keywords => {
if (keywords.size === 0) {
return null;
}
return new RegExp(filters.map(filter => {
let expr = escapeRegExp(filter.get('phrase'));
return new RegExp(keywords.map(keyword_filter => {
let expr = escapeRegExp(keyword_filter.get('keyword'));
if (filter.get('whole_word')) {
if (keyword_filter.get('whole_word')) {
if (/^[\w]/.test(expr)) {
expr = `\\b${expr}`;
}
@ -62,27 +62,15 @@ const regexFromFilters = filters => {
}).join('|'), 'i');
};
// Memoize the filter regexps for each valid server contextType
const makeGetFiltersRegex = () => {
let memo = {};
const getFilters = (state, { contextType }) => {
if (!contextType) return null;
return (state, { contextType }) => {
if (!contextType) return ImmutableList();
const serverSideType = toServerSideType(contextType);
const now = new Date();
const serverSideType = toServerSideType(contextType);
const filters = state.get('filters', ImmutableList()).filter(filter => filter.get('context').includes(serverSideType) && (filter.get('expires_at') === null || Date.parse(filter.get('expires_at')) > (new Date())));
if (!memo[serverSideType] || !is(memo[serverSideType].filters, filters)) {
const dropRegex = regexFromFilters(filters.filter(filter => filter.get('irreversible')));
const regex = regexFromFilters(filters);
memo[serverSideType] = { filters: filters, results: [dropRegex, regex] };
}
return memo[serverSideType].results;
};
return state.get('filters').filter((filter) => filter.get('context').includes(serverSideType) && (filter.get('expires_at') === null || filter.get('expires_at') > now));
};
export const getFiltersRegex = makeGetFiltersRegex();
export const makeGetStatus = () => {
return createSelector(
[
@ -90,10 +78,10 @@ export const makeGetStatus = () => {
(state, { id }) => state.getIn(['statuses', state.getIn(['statuses', id, 'reblog'])]),
(state, { id }) => state.getIn(['accounts', state.getIn(['statuses', id, 'account'])]),
(state, { id }) => state.getIn(['accounts', state.getIn(['statuses', state.getIn(['statuses', id, 'reblog']), 'account'])]),
getFiltersRegex,
getFilters,
],
(statusBase, statusReblog, accountBase, accountReblog, filtersRegex) => {
(statusBase, statusReblog, accountBase, accountReblog, filters) => {
if (!statusBase) {
return null;
}
@ -104,14 +92,17 @@ export const makeGetStatus = () => {
statusReblog = null;
}
const dropRegex = (accountReblog || accountBase).get('id') !== me && filtersRegex[0];
if (dropRegex && dropRegex.test(statusBase.get('reblog') ? statusReblog.get('search_index') : statusBase.get('search_index'))) {
return null;
let filtered = false;
if ((accountReblog || accountBase).get('id') !== me && filters) {
let filterResults = statusReblog?.get('filtered') || statusBase.get('filtered') || ImmutableList();
if (filterResults.some((result) => filters.getIn([result.get('filter'), 'filter_action']) === 'hide')) {
return null;
}
if (!filterResults.isEmpty()) {
filtered = filterResults.map(result => filters.getIn([result.get('filter'), 'title']));
}
}
const regex = (accountReblog || accountBase).get('id') !== me && filtersRegex[1];
const filtered = regex && regex.test(statusBase.get('reblog') ? statusReblog.get('search_index') : statusBase.get('search_index'));
return statusBase.withMutations(map => {
map.set('reblog', statusReblog);
map.set('account', accountBase);

1
app/javascript/packs/public.js

@ -4,6 +4,7 @@ import loadPolyfills from '../mastodon/load_polyfills';
import ready from '../mastodon/ready';
import { start } from '../mastodon/common';
import loadKeyboardExtensions from '../mastodon/load_keyboard_extensions';
import 'cocoon-js-vanilla';
start();

33
app/javascript/styles/mastodon/admin.scss

@ -915,7 +915,8 @@ a.name-tag,
text-align: center;
}
.applications-list__item {
.applications-list__item,
.filters-list__item {
padding: 15px 0;
background: $ui-base-color;
border: 1px solid lighten($ui-base-color, 4%);
@ -923,7 +924,8 @@ a.name-tag,
margin-top: 15px;
}
.announcements-list {
.announcements-list,
.filters-list {
border: 1px solid lighten($ui-base-color, 4%);
border-radius: 4px;
@ -976,6 +978,33 @@ a.name-tag,
}
}
.filters-list__item {
&__title {
display: flex;
justify-content: space-between;
margin-bottom: 0;
}
&__permissions {
margin-top: 0;
margin-bottom: 10px;
}
.expiration {
font-size: 13px;
}
&.expired {
.expiration {
color: lighten($error-red, 12%);
}
.permissions-list__item__icon {
color: $dark-text-color;
}
}
}
.dashboard__counters.admin-account-counters {
margin-top: 10px;
}

15
app/javascript/styles/mastodon/components.scss

@ -959,6 +959,21 @@
width: 100%;
clear: both;
border-bottom: 1px solid lighten($ui-base-color, 8%);
&__button {
display: inline;
color: lighten($ui-highlight-color, 8%);
border: 0;
background: transparent;
padding: 0;
font-size: inherit;
line-height: inherit;
&:hover,
&:active {
text-decoration: underline;
}
}
}
.status__prepend-icon-wrapper {

31
app/javascript/styles/mastodon/forms.scss

@ -1070,3 +1070,34 @@ code {
}
}
}
.keywords-table {
thead {
th {
white-space: nowrap;
}
th:first-child {
width: 100%;
}
}
tfoot {
td {
border: 0;
}
}
.input.string {
margin-bottom: 0;
}
.label_input__wrapper {
margin-top: 10px;
}
.table-action-link {
margin-top: 10px;
white-space: nowrap;
}
}

30
app/lib/feed_manager.rb

@ -352,7 +352,6 @@ class FeedManager
def filter_from_home?(status, receiver_id, crutches)
return false if receiver_id == status.account_id
return true if status.reply? && (status.in_reply_to_id.nil? || status.in_reply_to_account_id.nil?)
return true if phrase_filtered?(status, receiver_id, :home)
check_for_blocks = crutches[:active_mentions][status.id] || []
check_for_blocks.concat([status.account_id])
@ -388,7 +387,6 @@ class FeedManager
# @return [Boolean]
def filter_from_mentions?(status, receiver_id)
return true if receiver_id == status.account_id
return true if phrase_filtered?(status, receiver_id, :notifications)
# This filter is called from NotifyService, but already after the sender of
# the notification has been checked for mute/block. Therefore, it's not
@ -418,34 +416,6 @@ class FeedManager
false
end
# Check if the status hits a phrase filter
# @param [Status] status
# @param [Integer] receiver_id
# @param [Symbol] context
# @return [Boolean]
def phrase_filtered?(status, receiver_id, context)
active_filters = Rails.cache.fetch("filters:#{receiver_id}") { CustomFilter.where(account_id: receiver_id).active_irreversible.to_a }.to_a
active_filters.select! { |filter| filter.context.include?(context.to_s) && !filter.expired? }
active_filters.map! do |filter|
if filter.whole_word
sb = /\A[[:word:]]/.match?(filter.phrase) ? '\b' : ''
eb = /[[:word:]]\z/.match?(filter.phrase) ? '\b' : ''
/(?mix:#{sb}#{Regexp.escape(filter.phrase)}#{eb})/
else
/#{Regexp.escape(filter.phrase)}/i
end
end
return false if active_filters.empty?
combined_regex = Regexp.union(active_filters)
combined_regex.match?(status.proper.searchable_text)
end
# Adds a status to an account's feed, returning true if a status was
# added, and false if it was not added to the feed. Note that this is
# an internal helper: callers must call trim or push updates if

13
app/models/concerns/account_interactions.rb

@ -247,6 +247,19 @@ module AccountInteractions
account_pins.where(target_account: account).exists?
end
def status_matches_filters(status)
active_filters = CustomFilter.cached_filters_for(id)
filter_matches = active_filters.filter_map do |filter, rules|
next if rules[:keywords].blank?
match = rules[:keywords].match(status.proper.searchable_text)
FilterResultPresenter.new(filter: filter, keyword_matches: [match.to_s]) unless match.nil?
end
filter_matches
end
def followers_for_local_distribution
followers.local
.joins(:user)

87
app/models/custom_filter.rb

@ -3,18 +3,22 @@
#
# Table name: custom_filters
#
# id :bigint(8) not null, primary key
# account_id :bigint(8)
# expires_at :datetime
# phrase :text default(""), not null
# context :string default([]), not null, is an Array
# whole_word :boolean default(TRUE), not null
# irreversible :boolean default(FALSE), not null
# created_at :datetime not null
# updated_at :datetime not null
# id :bigint not null, primary key
# account_id :bigint
# expires_at :datetime
# phrase :text default(""), not null
# context :string default([]), not null, is an Array
# created_at :datetime not null
# updated_at :datetime not null
# action :integer default(0), not null
#
class CustomFilter < ApplicationRecord
self.ignored_columns = %w(whole_word irreversible)
alias_attribute :title, :phrase
alias_attribute :filter_action, :action
VALID_CONTEXTS = %w(
home
notifications
@ -26,16 +30,20 @@ class CustomFilter < ApplicationRecord
include Expireable
include Redisable
enum action: [:warn, :hide], _suffix: :action
belongs_to :account
has_many :keywords, class_name: 'CustomFilterKeyword', foreign_key: :custom_filter_id, inverse_of: :custom_filter, dependent: :destroy
accepts_nested_attributes_for :keywords, reject_if: :all_blank, allow_destroy: true
validates :phrase, :context, presence: true
validates :title, :context, presence: true
validate :context_must_be_valid
validate :irreversible_must_be_within_context
scope :active_irreversible, -> { where(irreversible: true).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()')) }
before_validation :clean_up_contexts
after_commit :remove_cache
before_save :prepare_cache_invalidation!
before_destroy :prepare_cache_invalidation!
after_commit :invalidate_cache!
def expires_in
return @expires_in if defined?(@expires_in)
@ -44,22 +52,55 @@ class CustomFilter < ApplicationRecord
[30.minutes, 1.hour, 6.hours, 12.hours, 1.day, 1.week].find { |expires_in| expires_in.from_now >= expires_at }
end
private
def irreversible=(value)
self.action = value ? :hide : :warn
end
def clean_up_contexts
self.context = Array(context).map(&:strip).filter_map(&:presence)
def irreversible?
hide_action?
end
def self.cached_filters_for(account_id)
active_filters = Rails.cache.fetch("filters:v3:#{account_id}") do
scope = CustomFilterKeyword.includes(:custom_filter).where(custom_filter: { account_id: account_id }).where(Arel.sql('expires_at IS NULL OR expires_at > NOW()'))
scope.to_a.group_by(&:custom_filter).map do |filter, keywords|
keywords.map! do |keyword|
if keyword.whole_word
sb = /\A[[:word:]]/.match?(keyword.keyword) ? '\b' : ''
eb = /[[:word:]]\z/.match?(keyword.keyword) ? '\b' : ''
/(?mix:#{sb}#{Regexp.escape(keyword.keyword)}#{eb})/
else
/#{Regexp.escape(keyword.keyword)}/i
end
end
[filter, { keywords: Regexp.union(keywords) }]
end
end.to_a
active_filters.select { |custom_filter, _| !custom_filter.expired? }
end
def prepare_cache_invalidation!
@should_invalidate_cache = true
end
def remove_cache
Rails.cache.delete("filters:#{account_id}")
def invalidate_cache!
return unless @should_invalidate_cache
@should_invalidate_cache = false
Rails.cache.delete("filters:v3:#{account_id}")
redis.publish("timeline:#{account_id}", Oj.dump(event: :filters_changed))
redis.publish("timeline:system:#{account_id}", Oj.dump(event: :filters_changed))
end
def context_must_be_valid
errors.add(:context, I18n.t('filters.errors.invalid_context')) if context.empty? || context.any? { |c| !VALID_CONTEXTS.include?(c) }
private
def clean_up_contexts
self.context = Array(context).map(&:strip).filter_map(&:presence)
end
def irreversible_must_be_within_context
errors.add(:irreversible, I18n.t('filters.errors.invalid_irreversible')) if irreversible? && !context.include?('home') && !context.include?('notifications')
def context_must_be_valid
errors.add(:context, I18n.t('filters.errors.invalid_context')) if context.empty? || context.any? { |c| !VALID_CONTEXTS.include?(c) }
end
end

34
app/models/custom_filter_keyword.rb

@ -0,0 +1,34 @@
# frozen_string_literal: true
# == Schema Information
#
# Table name: custom_filter_keywords
#
# id :bigint not null, primary key
# custom_filter_id :bigint not null
# keyword :text default(""), not null
# whole_word :boolean default(TRUE), not null
# created_at :datetime not null
# updated_at :datetime not null
#
class CustomFilterKeyword < ApplicationRecord
belongs_to :custom_filter
validates :keyword, presence: true
alias_attribute :phrase, :keyword
before_save :prepare_cache_invalidation!
before_destroy :prepare_cache_invalidation!
after_commit :invalidate_cache!
private
def prepare_cache_invalidation!
custom_filter.prepare_cache_invalidation!
end
def invalidate_cache!
custom_filter.invalidate_cache!
end
end

5
app/presenters/filter_result_presenter.rb

@ -0,0 +1,5 @@
# frozen_string_literal: true
class FilterResultPresenter < ActiveModelSerializers::Model
attributes :filter, :keyword_matches
end

24
app/presenters/status_relationships_presenter.rb

@ -2,7 +2,7 @@
class StatusRelationshipsPresenter
attr_reader :reblogs_map, :favourites_map, :mutes_map, :pins_map,
:bookmarks_map
:bookmarks_map, :filters_map
def initialize(statuses, current_account_id = nil, **options)
if current_account_id.nil?
@ -11,12 +11,14 @@ class StatusRelationshipsPresenter
@bookmarks_map = {}
@mutes_map = {}
@pins_map = {}
@filters_map = {}
else
statuses = statuses.compact
status_ids = statuses.flat_map { |s| [s.id, s.reblog_of_id] }.uniq.compact
conversation_ids = statuses.filter_map(&:conversation_id).uniq
pinnable_status_ids = statuses.map(&:proper).filter_map { |s| s.id if s.account_id == current_account_id && %w(public unlisted private).include?(s.visibility) }
@filters_map = build_filters_map(statuses, current_account_id).merge(options[:filters_map] || {})
@reblogs_map = Status.reblogs_map(status_ids, current_account_id).merge(options[:reblogs_map] || {})
@favourites_map = Status.favourites_map(status_ids, current_account_id).merge(options[:favourites_map] || {})
@bookmarks_map = Status.bookmarks_map(status_ids, current_account_id).merge(options[:bookmarks_map] || {})
@ -24,4 +26,24 @@ class StatusRelationshipsPresenter
@pins_map = Status.pins_map(pinnable_status_ids, current_account_id).merge(options[:pins_map] || {})
end
end
private
def build_filters_map(statuses, current_account_id)
active_filters = CustomFilter.cached_filters_for(current_account_id)
@filters_map = statuses.each_with_object({}) do |status, h|
filter_matches = active_filters.filter_map do |filter, rules|
next if rules[:keywords].blank?
match = rules[:keywords].match(status.proper.searchable_text)
FilterResultPresenter.new(filter: filter, keyword_matches: [match.to_s]) unless match.nil?
end
unless filter_matches.empty?
h[status.id] = filter_matches
h[status.reblog_of_id] = filter_matches if status.reblog?
end
end
end
end

9
app/serializers/rest/filter_keyword_serializer.rb

@ -0,0 +1,9 @@
# frozen_string_literal: true
class REST::FilterKeywordSerializer < ActiveModel::Serializer
attributes :id, :keyword, :whole_word
def id
object.id.to_s
end
end

6
app/serializers/rest/filter_result_serializer.rb

@ -0,0 +1,6 @@
# frozen_string_literal: true
class REST::FilterResultSerializer < ActiveModel::Serializer
belongs_to :filter, serializer: REST::FilterSerializer
has_many :keyword_matches
end

8
app/serializers/rest/filter_serializer.rb

@ -1,10 +1,14 @@
# frozen_string_literal: true
class REST::FilterSerializer < ActiveModel::Serializer
attributes :id, :phrase, :context, :whole_word, :expires_at,
:irreversible
attributes :id, :title, :context, :expires_at, :filter_action
has_many :keywords, serializer: REST::FilterKeywordSerializer, if: :rules_requested?
def id
object.id.to_s
end
def rules_requested?
instance_options[:rules_requested]
end
end

9
app/serializers/rest/status_serializer.rb

@ -13,6 +13,7 @@ class REST::StatusSerializer < ActiveModel::Serializer
attribute :muted, if: :current_user?
attribute :bookmarked, if: :current_user?
attribute :pinned, if: :pinnable?
has_many :filtered, serializer: REST::FilterResultSerializer, if: :current_user?
attribute :content, unless: :source_requested?
attribute :text, if: :source_requested?
@ -120,6 +121,14 @@ class REST::StatusSerializer < ActiveModel::Serializer
end
end
def filtered
if instance_options && instance_options[:relationships]
instance_options[:relationships].filters_map[object.id] || []
else
current_user.account.status_matches_filters(object)
end
end
def pinnable?
current_user? &&
current_user.account_id == object.account_id &&

26
app/serializers/rest/v1/filter_serializer.rb

@ -0,0 +1,26 @@
# frozen_string_literal: true
class REST::V1::FilterSerializer < ActiveModel::Serializer
attributes :id, :phrase, :context, :whole_word, :expires_at,
:irreversible
delegate :context, :expires_at, to: :custom_filter
def id
object.id.to_s
end