This might be the end of filter lists

Here goes a nice topic for my first post here :slight_smile:

From what I read on the chrome-extensions group, the new declarative webRequest API is inevitable. Despite the news that they backed off after the backlash, that does not mean they actually changed their plans.

The old webRequest API will be removed eventually so there will be no custom filtering engines anymore. Although, the declarative API will be improved compared to what we have in the draft.

I recommend you check out the current draft, the syntax looks like what we use for basic network filters so any filter list maintainer will easily understand it.

There are two important things about it:

  1. The ruleset is supposed to be hardcoded into the extension. This means that every filter list should be a different Chrome extension which is rather weird.
  2. To fix this, they added “dynamic rules” to the draft. This will allow ad blockers to compile a ruleset from different filter lists and then register it in runtime.

The problem here is that the number of rules in this dynamic ruleset cannot be greater than 5000 which is completely ridiculous. On a bright side, I believe that they will increase this limit, but I doubt they will let us have as many rules as we want.

In the worst case scenario, if they won’t increase the dynamic rules limit, there will be just a single filter list. Let’s not focus on this, though.

So what are the challenges for filter lists authors?

  1. Filter lists must become smaller. Redundant and obsolete rules must be removed, obviously.
  2. Filter lists might need to become more granular. Instead of “a filter against all ads” there should be a filter list against ads on top1000 websites or something like this.
  3. It might be that every ad blocker will need to have a main hardcoded filter list. For instance, ABP will likely use Easylist, uBO will use Easylist+uBlock filters, AdGuard will use Easylist+AdGuard Base filter. So when you create new rules, you’ll have to consider that there is a base list always enabled alongside your filter list.

What’s your take?

2 Likes

If we are to assume that there won’t be a way to edit the chrome.declarativeNetRequest.MAX_NUMBER_OF_* properties in about:config, then this would be an absolutely suicidal move of the Chromium team that would see Chrome lose 20% of its usershare inside of a month, and would make the userbases of e.g. Vivaldi so miniscule that they’d be discontinued. Chrome are clearly assuming that something that worked out for them on Android (To have no extensions at all) would work out on PC as well, but they don’t seem to know that PC users are a lot more technologically skilled than phone users.

We could very well end up in a situation where Firefox would become the world’s biggest browser for possibly the first time in recorded history.

Well, they said that they will increase these constants so it will not be as bad as it is now.

We will raise the rule limit from the draft 30K value. However, an upper limit is still necessary to ensure performance for users. Block lists have tended to be “push-only”, where new rules are added but obsolete rules are rarely, if ever, removed (external research has shown that 90% of EasyList blocking rules provided no benefit in common blocking scenarios). Having this list continue to grow unbounded is problematic.

then this would be an absolutely suicidal move of the Chromium team that would see Chrome lose 20% of its usershare inside of a month

Don’t be so sure. Look at Safari, they did impose similar limitations and disabled traditional extensions, and I see no loss of the market share.

Now that you mention it, EasyList has become very bloated over the past few years.

I hope no one with any connections to the Chromium team ever sees this comment of mine, but I have been very sporadically working on a sandbox test list of mine, which to the best of my knowledge has 90% functionality equivalence with EasyList with only 37 entries.

Not 37,000. Just 37.

But if I understand the consequences of removing webRequest from Chromium, then it’d also break hundreds of other extensions, like Tampermonkey, NoScript, Stylish, I Don’t Care About Cookies, and Redirector. So people who think that this will only affect adblockers, would be very disappointed when half of their extensions would become unsupported overnight.

And also the userbase of Safari on desktop was very small in the first place, so there were hardly anyone left who hadn’t left it already.

Now that you mention it, EasyList has become very bloated over the past few years.

Well, it’s not as bad. About 70% of EasyList rules are actually used. The problem is that 90% of these rules are used very rarely, on some unpopular websites.

But if I understand the consequences of removing webRequest from Chromium, then it’d also break hundreds of other extensions, like Tampermonkey, NoScript, Stylish, I Don’t Care About Cookies, and Redirector. So people who think that this will only affect adblockers, would be very disappointed when half of their extensions would become unsupported overnight.

  1. Stylish, I Don’t Care About Cookies and Redirector - should be okay.
  2. NoScript - might be broken.
  3. Tampermonkey/Violentmonkey/Greasemonkey - no technical issues with them, but they might be affected by the “remote scripts” limitation.
1 Like

And here is an update from Chrome developers.

What’s important:

  1. As I’ve said, declarative webRequest is what we’ll have to live with.
  2. However, the old API will still be available to enterprises (!).

By now I don’t know what the Chromium team have been smoking.

It feels like they used ABP (or a knockoff of ABP) for all of 3 minutes, using only the default settings, and then concluding that that was what the entirety of adblocking was about. They seem to not even know that there’s more than 1 filterlist in existence; in fact they don’t seem to even know about EasyList’s regional versions (like EasyList Germany).

Holy freaking cow. Why on earth are they trying to create a new extension API when they aren’t even using nor aware of any of their goddamn extensions?

If the Chromium team wants a war, then I’ll give them a war.

1 Like

Due to a few reasons listed below, the war has mostly been placed on hold for now:

  • I had misunderstood the dev’s comments about rolling out to “Canary” as meaning Chrome Canary instead of Chromium Canary.
  • https://9to5google.com/2019/05/29/chrome-ad-blocking-enterprise-manifest-v3/ was updated to show a note from Google about how they (according to themselves) were not intentionally sabotaging adblockers.
  • The EasyList maintainers’ belief (e.g. smed79) that list writers should remain neutral about things like these, thus I wouldn’t be able to rely on their support at all regarding this.
  • I’ll need to tend to other things in my life for the next week-and-a-half-ish, including a water leakage in my kitchen and to challenge myself to properly learn how to read German (beyond just kids media).

In theory, list authors will be able to easily create their own extensions with static lists of their filters instead of relying on ads blocking extensions.

This?

Chrome supports the use and development of ad blockers. We’re actively working with the developer community to get feedback and iterate on the design of a privacy-preserving content filtering system that limits the amount of sensitive browser data shared with third parties.

How this can be private if web requests can still be read?

While I am not aware of any of the details around making a static extension for each list maintainer, I can only imagine it’d be daunting for most maintainers to have to deal with Google’s add-on rules/community on a regular basis, in addition to the potential of having users’ extension drawers be cluttered beyond belief. A significant amount of maintainers (Me included) also couldn’t code extensions from scratch if it was a matter of life and death, so there’d have to be some easy extension templates available.

And that is indeed the quote I was thinking of.

How this can be private if web requests can still be read?

I suspect Google of not having any real idea what they’re doing, in addition to how I personally believe that the spokesperson behind the update note may potentially not actually be involved in the Manifest v3 process, even if it is semi-soothing words (s)he is speaking.

Yeah, but with a few issues:

  1. Cosmetic filters aren’t a part of this new API
  2. The lists will be completely independent, so:
    • You won’t be able to disable a rule from the other list. For instance, it sometimes necessary to disable and redefine some rules from EasyList.
    • Whitelisting a website will be the real hell. You’ll need to disable each one of these extensions.
    • Debugging lists will be quite hard for the same reason.
1 Like

Here is an update from Google: https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html

What’s important is that they’re planning to increase the limit to 150k. However, this does not solve the issue because they’re talking about the “static” rules limit (ruleset hardcoded into the extension).

We need the dynamic rules limit to be increased drastically from the current 5k value. Otherwise, having different filter lists is simply impossible.

Here’s a longer version:
https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/qFNF3KqNd2E/OnidEBO9BgAJ

2 Likes

Meanwhile, the discussion intensifies:
https://groups.google.com/a/chromium.org/d/msg/chromium-extensions/qFNF3KqNd2E/fJxfmDF-CAAJ

A member of Chrome’s security team tries to explain why the change is necessary and will improve performance and security.

IMO, the performance argument seems legit. What for the security argument, I am not convinced to say at least.

They, similar to me above, miss that half of the filter lists are cosmetic filters:

Injects CSS into a page.

To use this API you must have the permission for the page’s URL, either explicitly as a host permission, or using the activeTab permission.

Hmmm…

Also they think filter lists are created by little fairies. Filters are crowdsourced! Blocking extension must include logging feature to assist filter creators, and must be as easily available as possible. Even now people complain that uBO Logger is too complicated - developer tools will definitely not work for this.

Fortunately, this API is not going anywhere. But for using it we’ll have to request “any URL” permission, and that kinda defeats the purpose of DNR…

They promised a callback with the info on applied DNR rules.

This is a long shot beyond measure, but considering how Chrome has begun to show Adobe Flash shutdown notifications a year and a half early:

Is it a possibility that the Chromium team is trying to remove the previous manifest in late 2020 in an odd attempt to break support for Flash as hard as they physically can?

The only thing they need to do to achieve this is to not make much effort to support the old API:)