I agree, it's tedious and almost impossible to test the impacts of a change against all the possible configurations now. That's where CI is useful.
I'm testing the following framework :
- code and do some tests locally
- push to a "staging" branch
- this triggers the CI that runs the predefined tests and can build screenshots ( for docs, or review ) and packages ( including android ! )
- if the tests pass, the CI merges the changes into a master branch
This has a few advantages :
- if we add more tests we can code with more confidence
- we can know in less than 10 minutes after a commit if we broke something
- we can have binaries for each and every builds that succeed
- we can setup alpha and beta release in the Play store, instead of having a 3 years old version
- we can have a master branch with a code that is known to work since we merge only if the tests passed
- all the computing ressources for this are offloaded to a free 3rd party service, it avoids putting more stress on our own servers
- this is mostly based upon documented industry standards instead of using obscure, undocumented home made scripts

This has a few drawbacks :
- the current test process is based upon git, i can't get to something close using svn for now
- we need to spend some time implementing tests
Sleske, I'd be happy to get some help on this as this is a huge task. We can sync up on IRC or i can start a post / wiki page with more details if you want to contribute.