Haketilo

Haketilo moving away from being a browser extension seems like it would reduce the number of people who download it. Installing an extension is so easy (when it's signed), but every time I've used mitmproxy or other HTTP proxies, I've found it complicated to set up (though, I haven't used a proxy in a long time). Would it be possible, long term, to bundle the proxy as part of the web browser? I think it would be better to have an extension, even if it had limited functionality, but I don't know the details of how much effort that would take, so maybe it's not worth it.

Also, I found a page that seems to suggest that modifying CSP headers might be allowed in this case: https://discourse.mozilla.org/t/add-ons-policy-changes-2021-q-a/89654/1 (https://archive.ph/rkMEh or document.write(document.getElementsByTagName("noscript")[0].innerHTML); (Strangely view-source:https://discourse.mozilla.org/t/add-ons-policy-changes-2021-q-a/89654/1 seems to load forever.))

Q: Does " Add-ons must not relax web page security headers, such as the Content Security Policy" include addons that:

    Are designed to do specifically that - for example, CORS Everywhere,
    Add-ons that require relaxing security headers to be functional, such as an extension adding/modifying security headers (e.g. Referer) when accessing an API from background page which is normally inaccessible due to server-side CSRF protection or,
    a userscript loading extension using script tag/eval that replaces CSP?

A: Add-ons related to web development that are explicitly built and described to interact with those headers (e.g. CORS Everywhere) continue to be permitted. This is considered an edge case of the rule. Developers of such extensions should make an effort to clearly describe the security impact and ensure that users don’t accidentally have these security headers disabled while not actively using the add-on.
We’ve seen a number of add-ons that relax the CSP or remove certain headers out of convenience to enable certain functionality. This isn’t great because the protections website authors have put into place are being relaxed, which may have unintended consequences.
One example mentioned above I’d like to provide a bit more detail on, if you are setting a Referer header to achieve a specific result when making a singular request, this is not something disallowed by the policy. With this approach you are not relaxing security headers.

Have you asked Mozilla about this Q&A entry (not sure how the signing process works)? I'm not totally sure what is meant by "related to web development", so I guess Mozilla can deny it if they want. It would probably be good to have a mechanism by which the extension can be signed by another entity in addition to Mozilla, that browsers shipped with FSDG distros (Trisquel, etc.) trust in the way they currently trust Mozilla (Then again, maybe a waste of time if the extension is ultimately going away anyway).

Thanks for your feedback, Jacob. I respond inline.

Haketilo moving away from being a browser extension seems like it would reduce the number of people who download it. Installing an extension is so easy (when it's signed), but every time I've used mitmproxy or other HTTP proxies, I've found it complicated to set up (though, I haven't used a proxy in a long time).

I admit being able to install an extension is a great convenience. This is what made me start Haketilo as an extension in the first place. For some time, I was considering developing a proxy at a significantly later point, as a mere alternative to the extension.

Unfortunately, I am only a human and at some point I simply got discouraged by multiple things that didn't work as expected and slowed down the extension development. The AMO policy issue was just "the last straw that broke the camel's back". Perhaps the fact I gave up with the extension speaks badly of myself. Or perhaps it was actually a good decision? The policy issue is a rather significant one, after all, and there are loads of things that are only going to be achievable with a proxy.

If curious, you can look at some old issues like #15 to see what kinds of problems the WebExtensions platform used to cause.

Would it be possible, long term, to bundle the proxy as part of the web browser?

It would be. Although for that to happen, we'd need more people working on the project (the 2 contributors that I had are no longer participating 😢).

Have you asked Mozilla about this Q&A entry (not sure how the signing process works)?

No, I haven't. In fact, I did not see it before. I guess Haketilo would fall into the category of "add-ons that relax the CSP or remove certain headers out of convenience to enable certain functionality", but I might actually ask them about it (just not right now).

I guess Mozilla can deny it if they want

They surely can. This is yet another thing I don't like - that even though the software (well, 99% of it) is nominally free/libre, I might still get trapped by it.

It would probably be good to have a mechanism by which the extension can be signed by another entity in addition to Mozilla, that browsers shipped with FSDG distros (Trisquel, etc.) trust in the way they currently trust Mozilla (Then again, maybe a waste of time if the extension is ultimately going away anyway).

In fact, I am surprised something like this has not yet been done. Ideally, the browser should allow users to specify what entities they want to trust. And this would make sense regardless of Haketilo being there or not :)

I think it would be better to have an extension, even if it had limited functionality, but I don't know the details of how much effort that would take, so maybe it's not worth it.

You might have just pointed out to the main problem of mine: that I am trying to make my product perfect and spend a lot of time. If I instead delivered something that's just "good enough" but did it faster, it could be way better for the actual project.

The amount of effort needed to battle the WebExtension APIs has so far been enormous. You know what? While I cannot make the AMO policy issue go away, I have a vision of how we could at least make the extension development more approachable:

1. We could make it Firefox-only. Chromium releases were not receiving much interest and other browsers would be covered by the proxy anyway. At the same time, Firefox APIs have more to offer (StreamFilter, blocking webRequest under manifest v3).
2. We could restrict the extension to only working with HTTP(s) pages. Support for file:// and ftp:// (for older browsers that still supported it) is of less importance and diverges the development from focusing on what's really important.
3. Once we agree for 1. and 2., we can handle all pages with just StreamFilter and webRequest which simplifies things A LOT.

I am still unsure how best to comply with the other problematic AMO policy point:

Only release versions of third-party libraries and/or frameworks may be included with an add-on. Modifications to these libraries/frameworks are not permitted.

This by itself sounds like an imprisonment of free software. And it's of course incompatible with the current Haketilo build system... We could move away from our custom build system to some established one at the cost of adding some build-time dependencies (up until now Haketilo could be build with just POSIX). Actually, doing this earlier could have saved me some effort (mea culpa).

We'd also need to find a way to make automated tests easier to maintain. So far I've been using pytest and selenium. The first problem is that I had to put a lot of JS code inside Python strings which is inconvenient. I also didn't make the "unit tests" (quotes needed) truly restricted to single functions which led to complex test cases (this is my own fault again). The error messages from selenium are also rather poor and the tests themselves take somewhat long to run 😕

A reasonable solution might be to test the JS code outside of browser, for example with a library like JSDOM (also packaged for Debian ^^).

I haven't yet used the alternative technologies I am considering here. I'd of course need to spend additional time learning them. It might be worth it, though.

So, I gave you the relevant details. You can share your thoughts on them. Just please keep in mind that I'm not likely to refocus on the extension until I actually have some other contributors to help me with it.

Lastly, I am sorry for taking so long to respond (Redmine's email notifications seem to have failed me once again)

every time I've used mitmproxy or other HTTP proxies, I've found it complicated to set up

I might have a good news for you: it should be perfectly possible to package the proxy into a .deb so that it starts automatically and automatically installs its certificate system-wide :)

When I posted my first comment, I noticed it didn't show up in the activity page, so I thought maybe you wouldn't get an email. For some reason it didn't occur to me that I would not get an email when you replied haha. I clicked the "Watch" button so maybe I'll get an email about new comments now.

I think the idea to make the extension Firefox-only and only support HTTP(s) pages is a good idea. Since the extension is already made and works, it seems like all you'd have to do in the future is fix bugs and compatibility with newer browser versions that don't exist yet. Now that Haketilo scripts can bypass CORS, is there any page that can't be fixed with Haketilo as it is now? Regardless, I think it's fine for new methods in the extension to be a low priority.

Only release versions of third-party libraries and/or frameworks may be included with an add-on. Modifications to these libraries/frameworks are not permitted.

I wonder how this policy point deals with libraries that are based on other libraries? Regardless this seems like a lot of effort to work around. I wonder what the justification for this requirement is? Perhaps the reviewers want to be able to say library Av1.0.2 is good rather than evaluating many different variations of A made by independent developers.

At some point, it seems Trisquel updated Abrowser so that it no longer requires signatures on extensions, now showing a warning instead of completely blocking installation. Or maybe I accidentally changed something on my machine only.

I clicked the "Watch" button so maybe I'll get an email about new comments now.

Unfortunately, it seems to have no effect :/

it seems like all you'd have to do in the future is fix bugs and compatibility with newer browser versions that don't exist yet

This is literally true. Yet, some bugs might require enormous amount of work to fix (#115). And maintaining compatibility once Firefox switches to MV3 could also require substantial effort.

I can still make bug & compatibility fixes as long as they are simple ones. For anything greater, I'd need more resources. That is, either code contributors or some funding because the NLnet fund is approaching its end (with only a small part of its goals completed, I admit with sadness while blaming WebExtension APIs 😉).

Now that Haketilo scripts can bypass CORS, is there any page that can't be fixed with Haketilo as it is now?

Yup. Or rather, there are libraries that cannot be packaged for Haketilo because they rely on dynamic loading of their components. MathJax is an example that actually causes problems (#114).

I wonder how this policy point deals with libraries that are based on other libraries? Regardless this seems like a lot of effort to work around.

I had similar thoughts. Although I'm not convinced that making a simple rebranded fork is "a lot of effort".

Anyway, perhaps instead of fighting with AMO's rules we should just spin up our own extension store as suggested to me by one Haketilo user? And try convincing the devs of Abrowser, LibreWolf, etc. to include it in their browsers?

Just a thought. I am not going to take this up anytime soon.

At some point, it seems Trisquel updated Abrowser.

Indeed

Oh, I just found the relevant email notification settings in Redmine. It might work now

Now that Haketilo scripts can bypass CORS, is there any page that can't be fixed with Haketilo as it is now?

Yup. Or rather, there are libraries that cannot be packaged for Haketilo because they rely on dynamic loading of their components. MathJax is an example that actually causes problems (#114).

Another thing that cannot be done yet is spoofing of the "Referer" header. If we could do that, we'd also be able to - for example - write Hacktcha as a simple library so that it works without opening a reCAPTCHA page inside an iframe. It doesn't seem like something important... But because of that missing feature it is currently impossible to use Hacktcha on one page and normal reCAPTCHA on another page in the same browser profile. When Hacktcha is installed for one page, it breaks normal reCAPTCHA everywhere else :/

Some sites might also be non-fixable because we lack the ability to manage http-only cookies from Haketilo scripts