I think you’re right that we need to have this trigger as an automatic check. It should be part of the theme activation process. There are a few hooks since WP 3.x for “on theme activation” or “on theme change” that give different types of functionality.
If we used “after_switch_theme” it should only fire once, at the time the theme is activated. I believe one of the variables passed in this function is the theme name so we could test if “runway” is being activated to make sure we aren’t running this every time a child theme is called.
There is also the “after_setup_theme” function which is more generic. It runs every time the theme loads so we would be constantly testing for these things. I’m of the opinion it should be a one time check then the Report Manager will come into play alerting the user if a test fails.
Running it once with “after_switch_theme” is probably best. If you get a message of an issue after this point it might be the case of a new child theme being installed/created, a user moving their themes to another WP install or a folder manually deleted. This seems to cover the intended purpose and allows users to manage their files themselves but still receive a notification of any potential problems. I know I wouldn’t like it if I deleted the “download_dir” folder manually only to have Runway instantly recreate it instantly. There might be a good reason for deleting it or some testing taking place. Having it created in the first place is the only obligation here in my opinion.